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VALIDATION  OF  CREW  COORDINATION  TRAINING  AND  EVALUATION 
METHODS  FOR  ARMY  AVIATION 


Introduction 

Dynamics  Research  Corporation  (DRC) ,  under  contract  to  the 
Army  Research  Institute  Rotary-Wing  Aviation  Research  Unit 
(ARIRWARU) ,  developed  and  evaluated  an  exportable  Army  Crew 
Coordination  training  and  evaluation  package.  The  development  of 
the  training  and  evaluation  systems  is  discussed  in  companion 
voliomes  delivered  to  ARIRWARU  (Grubb,  Simon,  Leedom,  &  Zeller,  in 
preparation;  and  Pawlik,  Simon,  Grubb,  &  Zeller,  in  preparation) . 
Two  products,  a  Field  Exportable  Evaluation  Package  (Grubb, 

Simon,  &  Zeller,  1992)  and  a  Field  Exportable  Training  Package 
(Pawlik,  Simon,  Grubb,  &  Zeller,  1992),  were  also  delivered  under 
the  current  contract.  The  U.S.  Army  Aviation  Center  (USAAVNC) 
sponsored  a  crew  coordination  validation  testbed  of  DRC^s 
materials  and  methods  at  Fort  Campbell,  KY  from  3  August  -  2 
September  1992.  This  report  focuses  on  the  results  of  using  the 
evaluation  measures  (Grubb  et  al.,  in  preparation)  during  that 
testbed.  See  Figure  1,  crew  coordination  evaluation  and  training 
products . 

Background 

For  the  past  three  years,  ARIRWARU  has  undertaken  a 
comprehensive  research  and  development  effort  on  crew 
coordination  training  and  evaluation  in  Army  aviation.  A  proof- 
of-concept  training  and  evaluation  system  was  first  tested  with 
two  assault  helicopter  battalions  (UH-60)  at  Fort  Campbell  during 
the  Spring  of  1990  (Simon,  1990,  1991).  At  about  the  same  time, 
the  U.S.  Army  Safety  Center  (USASC)  identified  crew  coordination 
failures  as  a  major  problem.  During  the  period  FY  84-89,  crew 
coordination  failures  identified  by  the  USASC  directly 
contributed  to  the  loss  of  147  lives  and  $292  million  in  aviation 
accident  costs.  These  accidents  were  caused  by  the  failure  of 
experienced  and  qualified  aviators  to  coordinate  their  decisions 
and  actions  in  the  cockpit.  In  1990,  the  Deputy  Chief  of  Staff 
for  Operations  and  Plans  directed  the  USAAVNC  to  focus  its 
aviation  training  and  evaluation  on  crew  performance.  In  1991, 
ARIRWARU  assisted  the  USAAVNC  in  the  rewriting  of  TC  1-210, 
Aircrew  Training  Program  Commander's  Guide  to  Individual  and  Crew 
Training,  and  various  Aircrew  Training  Manuals  (ATM)  for  each 
aircraft  type.  Through  these  efforts,  the  Army  was  able  to 
define  more  precisely  the  need  for  crew  coordination  and  the 
means  for  achieving  it.  Much  of  this  unprogrammed  assistance  was 
based  on  ARIRWARU 's  research  on  crew  error  patterns. 
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Figure  1.  Summary  and  relationships  of  crew  coordination 
evaluation  and  training  products. 


By  1992,  it  was  clear  that  additional  training  research 
would  be  needed  to  determine  how  best  to  transition  existing 
aircrews  to  the  new  training  standards.  As  a  result,  the  USAAVNC 
sponsored  ARIRWARU  to  develop  and  validate  a  new  exportable 
training  concept  for  Army  aviators.  While  some  training 
materials  were  available  from  commercial  aviation  and  other 
services,  their  contents  were  not  suited  to  Army  aviation. 
Additional  challenges  associated  with  this  training  research 
included  (a)  identification  of  specific  human  error  patterns  that 
were  correctable  through  training,  (b)  development  of  observable 
performance  standards,  and  (c)  identification  of  crew  member 
actions  required  for  each  specific  flight  task. 

Beginning  in  February  1992,  DRC  worked  closely  with  the 
USAAVNC  Working  Group  to  review  existing  materials  and  develop  a 
training  and  evaluation  system  to  accompany  the  newly  revised 
ATMs.  The  Working  Group  guided  DRC  in  defining  key  concepts  for 
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the  system,  such  as  the  Crew  Coordination  Objectives  and  the 
related  Basic  Qualities,  that  crews  should  master  during  training 
and  demonstrate  during  evaluations.  After  drafting  training  and 
evaluation  materials  and  methods  for  a  crew  coordination 
validation  testbed  effort,  DEC  worked  with  the  101st  Aviation 
Brigade  (Air  Assault)  at  Fort  Campbell  to  implement  the 
validation  testbed.  The  objectives  of  the  validation  testbed 
were  to: 


1.  Demonstrate  and  validate  the  new  field  exportable 
program  for  training  and  evaluating  crew  coordination  skills  in 
Army  aviators. 

2.  Assess  the  potential  for  this  type  of  training  to 
increase  military  effectiveness  and  enhance  aviation  safety. 

Methodology  and  Sample  Description 


Methodology 

Drafts  of  the  Field  Exportable  Training  Package  (Pawlik  et 
al.,  1992)  and  the  Field  Exportable  Evaluation  Package  (Grubb  et 
al.,  1992),  and  the  implementation  methods  associated  with  them, 
were  tested  at  Fort  Campbell.  These  packages  are  briefly 
summarized  below: 

Aircrew  Coordination  Exportable  Training  Package  (Pawlik  et 
al.,  1992)  -  This  training  package  was  designed  for  three 
audiences:  trainers,  instructors,  and  aircrews.  The  USAAVNC- 

certified  crew  coordination  trainers  teach  unit  instructors,  who 
then  train  unit  aircrews  (rated  and  nonrated  crew  members) . 
Training  for  the  unit  instructors  in  the  testbed  version  of  the 
aircrew  coordination  course  consisted  of  26  hours  of  instruction 
broken  into  four  phases  covering  methods  of  instruction,  the 
Aircrew  Coordination  Student  Course,  an  evaluation  workshop,  and 
scenario  familiarization.  Training  for  unit  aircrew  testbed 
participants  consisted  of  18  classroom  hours  and  two  5-hour 
training  missions  in  the  visual  flight  simulator  facility.  The 
5-hour  simulator  sessions  included  premission  planning,  simulator 
flight,  postmission  after-action  review  conducted  by  the 
aircrews,  and  an  instructor  debrief  of  the  entire  process. 

The  goal  of  the  initial  training  was  to  bring  the 
participating  aircrews  to  a  satisfactory  level  of  performance  of 
crew  coordination  skills  and  behaviors.  It  was  expected  that 
performance  above  the  satisfactory  level  would  take  more  training 
time  than  was  available  during  the  testbed. 

Crew  Coordination  Exportable  Evaluation  Package  for  Army 
Aviation  (Grubb  et  al.,  1992)  -  This  field  exportable  evaluation 
package  was  designed  for  unit  instructors  (instructor  pilots 
(IPs)  and  unit  trainers  (UTs) )  to  complement  and  evaluate  the 
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training  provided  to  unit  aircrews.  Several  measures  were 
designed  specifically  for  use  in  the  validation  testbed  and  are 
not  intended  to  be  fielded.  The  primary  question  asked  through 
the  use  of  the  evaluation  measures  was  whether  the  aircrews 
showed  improvement  between  the  pre-  and  post-training 
evaluations.  The  secondary  questions  were  whether  the  IPs  easily 
understood  and  used  the  measures,  and  whether  the  measures 
produced  reliable  evaluation  data.  The  measures  and  the  results 
of  their  use  in  the  testbed  are  described  in  their  respective 
sections  within  this  report.  Following  is  the  list  of  measures 
composing  the  measurement  suite  used  in  the  testbed.  Measures 
that  are  part  of  the  exportable  evaluation  package  are 
italicized. 

Attitude  —  The  "Army  Aviation  Crew  Member  Questionnaire” 

Behavior  —  Basic  Qualities  associated  with  crew 
coordination  captured  through  use  of  the  Aircrew  Coordination 
Evaluation  (ACE)  Checklist 

Behavior  and  Performance  —  A  grade  slip  based  on  UH-60  ATM 
Tasks  revised  to  incorporate  aircrew  coordination  considerations 

Crew  Performance  —  Measures  of  crew  performance  made  during 
tactical  scenario  execution 

Participant  Debriefings  —  Form  for  debriefing  all  testbed 
participants  at  the  conclusion  of  the  testbed. 

Scenario  descriptions.  Two  utility  helicopter  tactical 
missions  were  used  for  the  crew-level  evaluations  and  to  assess 
changes  in  crew  mission  performance.  The  baseline  evaluation  was 
conducted  prior  to  the  crew  coordination  training  (pre-training 
condition) .  The  second  evaluation  was  administered  after  the 
training  (post-training  condition) .  The  two  missions  were  very 
similar  in  difficulty  in  terms  of  time  stress,  navigational 
demands,  quantity  and  capabilities  of  simulated  threat,  etc.  The 
objectives  and  tasks  incorporated  into  the  scenario  were  all  made 
to  present  two  equally  difficult  missions  to  the  aircrews. 

Testbed  participants.  Four  IPs  and  four  UTs  received 
training  to  present  the  Aircrew  Coordination  Student  Course  and 
to  perform  the  pre-  and  post-training  evaluations.  Sixteen  two- 
person  aircrews  were  battle-rostered  for  the  testbed.  Even 
though  the  sample  size  for  the  testbed  was  rather  small, 
statistically  significant  results  were  obtained  on  many  of  the 
variables  investigated.  This  means  that  stable  estimates  of  what 
will  happen  in  the  larger  population  of  Army  aviators  in  terms  of 
changes  from  the  pre-  to  post-training  condition  were  obtained. 

As  will  be  shown,  these  results  speak  very  positively  of  the 
military  worth  of  the  crew  coordination  training  program. 
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All  of  the  testbed  participants  listed  UH-60  as  their 
primary  aircraft  and  were  drawn  from  the  5th  and  9th  Battalions 
of  the  101st  Aviation  Brigade  (Air  Assault)  and  the  2nd  Squadron, 
17th  Cavalry  Regiment.  Thirty-one  of  the  32  aviators  had  a 
Readiness  Level  1  rating ;  one  aviator  was  Readiness  Level  2 .  The 
participants'  experience  level  was  sufficiently  broad.  Table  1 
presents  a  description  of  the  testbed  participants. 


Table  1 

Rank  and  Experience  Level  of  Testbed  Participating  Aviators 


Instructors 

Aviators 

Rank 

ILT 

0 

2 

CW3 

1 

0 

CW2(P) 

3 

0 

CW2 

4 

15 

WOl 

15 

UH-60  experience 

Range 

500-2300  hrs 

70-1850  hrs 

Mean 

1219  hrs 

513  hrs 

Time  in  unit 

Range 

1-37  mos 

1-48  mos 

Mean 

19  mos 

15  mos 

Testbed  schedule.  The  testbed  schedule  is  graphically 
presented  in  Figure  2.  The  first  step  in  the  testbed  was  for 
project  staff  trainers  to  instruct  the  participating  IPs  and  UTs. 
After  receiving  instruction,  the  IPs  (not  UTs)  rated  the  16 
battle-rostered  participating  aircrews  during  a  full  (premission, 
flight,  and  postmission)  simulator  session.  This  was  the  "pre- 
training”  evaluation  designed  as  the  baseline  against  which 
performance  improvements  would  be  measured.  The  16  aircrews  were 
divided  into  two  groups  of  8  for  the  classroom  instruction.  Two 
teams  each  consisting  of  two  IPs  and  two  UTs  were  formed  to  team- 
teach  the  classroom  instruction.  The  instructor  teams  also 
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instructed  the  aircrews  during  the  two  course-related  simulator 
training  missions.  It  should  be  noted  that  the  IP/UT  cadres 
prepared  for  the  classroom  presentations  and  developed  the 
simulator  scenarios  for  the  training  missions  during  very  busy 
schedules;  they  were  not  relieved  of  operational  reguirements 
during  the  month-long  testbed.  This  was  a  very  demanding  period 
for  them.  Subseguent  to  the  training,  another  evaluation 
mission,  the  post-training  evaluation,  was  given  to  each  of  the 
16  aircrews.  The  eight  crews  given  scenario  one  for  the  pre- 
training  mission  were  given  scenario  two  for  the  post-training 
mission  and  vice  versa.  Again,  crews  were  rated  using  the 
measures  described  above.  When  the  evaluation  missions  were 
completed,  all  testbed  participants  were  debriefed  on  the  testbed 
and  asked  to  critique  the  training. 

Overview  of  Report 

The  remainder  of  this  report  is  divided  into  four  major 
sections.  The  "Results  of  the  Testbed"  section  briefly  discusses 
each  of  the  measures  used  in  the  validation  testbed  and  the 
results  for  the  pre-training  and  post-training  evaluation 
missions.  Where  appropriate,  frequency  tables  of  the  results  of 
administering  each  measure  are  provided  in  an  appendix. 

The  "Operational  Safety  Implications"  section  is  divided 
into  two  parts.  The  first  part  is  a  discussion  of  the  impact  of 
the  validation  testbed  on  "marginal"  crews,  that  is  crews  who 
performed  particularly  poorly  during  the  pre-training  evaluation 
mission.  The  second  part  is  a  discussion  of  the  potential^ 
relationship  between  the  data  collected  during  the  validation 
testbed  and  other  Army-sponsored  studies. 

The  last  two  sections  present  recommendations  for  fielding 
the  crew  coordination  training  and  evaluation  packages, 
recommendations  for  future  research  in  the  area  of  crew 
coordination,  and  several  issues  on  the  future  of  the  USAAVNC's 
crew  coordination  project. 

Results  of  the  Testbed 

This  section  describes  each  measure  used  in  the  validation 
testbed  and  the  results  from  the  pre-  and  post-training 
administration  of  the  measures. 

The  Army  Aviation  Crew  Member  Questionnaire 

The  questionnaire  administered  to  the  testbed  participants 
has  its  origins  in  the  commercial  world;  it  has  since  been 
revised  several  times. 
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Figure  2.  Testbed  activities  schedule. 


The  commercial  aviation  questionnaire,  the  "Cockpit 
Management  Attitudes  Questionnaire"  (CMAQ) ,  (Gregorich, 

Helmreich,  &  Wilhelm,  1990)  was  modified  for  use  by  the  Army. 

This  questionnaire  is  also  referred  to  as  the  "Army  CMAQ:  (Simon, 
et  al.,  1992).  Although  "CMAQ"  is  a  term  widely  understood  by 
DoD  and  commercial  aviation  evaluation  research  specialists,  the 
project  staff  believed  that  a  more  innocuous  name  for  the 
attitude  questionnaire  would  be  desirable,  hence  its  current 
name,  the  Army  Aviation  Crew  Member  Questionnaire. 

The  Army  Aviation  Crew  Member  Questionnaire  is  slightly 
different  than  the  one  Simon  reported  in  1992;  several  questions 
were  revised  and  others  added  based  on  lessons  learned  from 
previous  data  collection  efforts.  The  current  version  includes 
46  statements  for  which  aviators  are  asked  to  rate  the  extent  of 
their  agreement  or  disagreement  on  a  seven-point  scale.  Also 
included  in  the  questionnaire  is  a  section  used  to  collect 
background  data  from  the  respondents.  The  current  version  of  the 
Army  Aviation  Crew  Member  Questionnaire  is  presented  in  Appendix 
A. 


The  Army  Aviation  Crew  Member  Questionnaire  is  used  to 
assess  three  primary  attitudes  associated  with  crew  coordination. 
Table  2  presents  the  names,  definitions,  and  example  items  for 
each  attitude  area  assessed. 

Results  of  the  attitude  cmestionnaire.  Item-by-item  results 
in  the  form  of  frequency  tables  for  the  items  composing  the  Army 
Aviation  Crew  Member  Questionnaire  are  presented  in  Appendix  A. 
The  questionnaire  was  administered  twice:  once  at  the  end  of  the 
pre-training  evaluation  mission  (baseline)  and  a  second  time  at 
the  end  of  the  post-training  evaluation  mission.  Thus,  each  item 
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has  two  frequency  tables  associated  with  it:  one  for  the  pre- 
and  one  for  the  post-training  evaluation.  As  can  be  seen, 
although  the  Army  aviators  in  our  sample  generally  had  a 
favorable  attitude  toward  crew  coordination  in  both  the  pre-  and 
post-training  conditions,  post-training  attitudes  were  usually 
more  favorable. 


Table  2 

Definitions  and  Example  Items  for  the  Army  Aviation  Crew  Member 
Questionnaire _ _ 

Subscale  name  and  definition  Example  items 


Communication  &  Coordination  • 

An  orientation  toward 
interpersonal  awareness, 
communication,  and  crew 
coordination  • 


Shared  Leadership  • 

An  attitude  toward  the 
appropriateness  of  sharing 
responsibility  for  leadership 


Recognition  of  Stressor  Effects  • 
An  attitude  accepting  that 
human  performance  is  affected 
by  external  events  and  • 

allowance  must  be  made  for 
changed  performance 


Good  crew  communication  and  crew 
coordination  are  as  important  as 
technical  proficiency  for  the  safety  of 
the  flight. 

The  pilot- in- command  should  use  his 
crew  to  help  him  maintain  situation 
awareness. 

Pilots-in-command  who  accept  and 
implement  suggestions  from  the  crew 
lessen  their  stature  and  reduce  their 
authority. 

Crew  members  should  be  able  to 
anticipate  requirements  as  the  mission 
progresses. 

Crew  Member  task  overload  usually 
occurs  because  the  crew  member  is  not 
very  competent. 

Each  crew  member  should  watch  for 
situations  in  which  external  events 
limit  others '  performance. _ 


To  capture  more  precisely  the  differences  in  attitudes 
between  the  pre-  and  post-training  conditions,  four  scales  were 
created.  Scale  scores  were  computed  for  the  3  attitude  areas 
assessed  by  the  Army  Aviation  Crew  Member  Questionnaire,  and  a 
total  score  was  computed  for  all  46  items.  Several  steps  were 
taken  to  compute  the  scores.  First,  negatively  worded  items  were 
recoded.  Twenty-three  items  were  worded  in  the  negative  such 
that  a  response  indicating  disagreement  was  the  desirable  or 
"correct”  response  and  a  response  indicating  agreement 
represented  an  undesirable  attitude.  Each  of  these  items  was 
recoded  so  that  1=7,  2=6,  3=5,  5=3,  6=2,  and  7=1.  Table  3  shows 
the  items  in  each  of  the  subscales  and  the  direction  of  the 
desired  response. 
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Table  3 


Items  Composing  the  Three  Attitude  Scales  with  Notations®  for 
Negatively  Worded  Items _ 


Army  Aviation  Crew  Member 

Questionnaire  scale  name _  Items  in  the  scale 


Communication  &  Coordination 

Shared  Leadership 

Recognition  of  Stressor  Effects 
Whole  Questionnaire  _ 


1,  2,  3,  4,  5^  7,  9,  11,  12,  16,  20,  21, 
22,  24,  27,  28,  31®,  35,  37®,  38®,  41®,  42 

8®,  13®,  17®,  18®,  19®,  23,  26®,  29®,  30®, 
33®,  34®,  36®,  40®,  44,  46 

6®,  10,  14®,  15®,  25®,  32®,  39,  43®,  45® 

1  through  46 _ 


®The  desired  response  for  these  items  was  in  the  direction  of  "disagree." 
They  were  recoded  so  that  the  desired  response  would  appear  positive;  thus, 
scale  comparisons  could  be  made  because  higher  scores  denoted  better 
attitudes. 


Cronbach's  alpha  (a  reliability  coefficient  based  on 
internal  consistency)  was  computed  for  each  scale.  The 
reliability  coefficients  for  the  scales  were:  Communication  & 
Coordination,  .76;  Shared  Leadership,  .59;  Recognition  of 
Stressor  Effects,  .52;  and  for  the  entire  questionnaire,  .76. 
Although  these  coefficients  are  only  moderately  good,  they  are 
similar  to  those  obtained  with  the  commercial  version  of  the  CMAQ 
(Simon  et  al.,  1992). 

Next,  average  item  scores  for  each  scale  were  computed  for 
the  pre-  and  post-training  conditions.  The  scale  scores  were 
compared  using  a  paired  t-test  to  determine  whether  the 
difference  was  significantly  different  from  one  administration  to 
the  next.  The  robustness  of  the  parametric  t-test  was  considered 
in  selecting  the  statistical  method  for  analyzing  the  ordinal 
scale  data.  Table  4  shows  the  means  of  the  pre-  and  post¬ 
training  administrations  and  whether  the  difference  between  the 
two  is  statistically  significant. 

There  was  an  attitudinal  improvement  between  the  pre-  and 
post-training  evaluation  sessions.  The  Communication  and 
Coordination  scale  resulted  in  less  change  than  the  other  scales. 
There  may  be  several  explanations  for  this  small  difference.  It 
may  be  that  the  Army  currently  teaches  some  of  the  values 
associated  with  Communication  and  Coordination  in  other  courses. 
Perhaps  the  testbed  aviators  were  "test-smart”  about  these  items 
and  tried  to  second  guess  the  answer  they  thought  was  the 
desirable  or  respectable  attitude.  It  is  also  noteworthy  that 
the  Communication  and  Coordination  subscale  had  the  highest  ^ 
average  scores;  thus,  a  statistical  "ceiling  effect”  may  exist ^ 
for  this  subscale.  In  any  event,  the  change  for  the  subscale  is 
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Table  4 


Comparison  of  Mean  Item  Scores  for  Pre-  and  Post-training 
Administrations  of  the  Army  Aviation  Crew  Member  Questionnaire 


Attitude  scale 

Pre¬ 

training 

Post¬ 

training 

Communication  and  Coordination 

5.97 

6.07 

Shared  Leadership 

5.57 

icic 

5.78 

Recognition  of  Stressor  Effects 

4.51 

* 

4.70 

Whole  Questionnaire 

5.55 

** 

5.71 

*E  <  .05.  **E  <  .01.^ 


in  the  right  direction  so,  at  a  minimum,  the  training  served  to 
reinforce  existing  attitudes.  The  other  two  subscales.  Shared 
Leadership  and  Recognition  of  Stressor  Effects,  showed 
significant  positive  improvement  from  the  pre-  to 
postadministrations .  It  is  thought  that  because  these  notions 
may  be  relatively  novel  for  the  testbed  aviators,  and  because  the 
training  presented  new  conceptual  information,  the  aviators  did 
not  try  to,  or  could  not,  "second-guess"  the  desired  response. 
Furthermore,  there  was  less  of  a  ceiling  effect  on  these  two 
subscales . 

An  understanding  of  the  concept  of  Shared  Leadership 
requires  a  mature  crew  member  who  willingly  assumes 
responsibility  for  mission  safety  and  success  regardless  of 
whether  he  is  the  pilot-in-command,  the  pilot,  or  the  flight 
engineer.  This  individual  is  not  concerned  about  preserving 
personal  rank  or  status  as  much  as  he  is  concerned  with  mission 
effectiveness  and  safety.  He  also  considers  his  fellow  crew 
members  to  be  an  important  resource  and  that  they  must  be 
respected  and  used  as  valued  members  of  the  team. 


^For  this  table  and  subsequent  tables,  significant  change  * 
for  E  <  *05  and  **  for  e  <  ‘Oi*  This  means  that  if  we  were  to 
run  a  testbed  under  similar  conditions  again,  the  difference 
between  the  pre-training  score  and  the  post-training  score  would 
be  as  large  as  the  differences  found  at  Fort  Campbell  95  out  of 
100  times  or  99  out  of  100  times. 

The  reader  is  encouraged  not  to  confuse  statistical 
significance  with  real-world  significance.  Statistical 
significance  deals  with  the  repeatability  of  the  findings. 
Practical  (real-world)  significance  has  to  do  with  whether  the 
findings  will  make  a  difference  to  real  world  users. 
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An  understanding  of  the  notion  of  Recognition  of  Stressor 
Effects  requires  an  aviator  who  knows  that  neither  he  nor  his 
fellow  crew  members  are  superhuman.  This  individual  knows  that 
human  errors  are  a  fact  of  life  and  that  everyone  makes  mistakes 
from  time  to  time;  he  knows  that  errors  should  be  corrected  with 
a  minimum  of  disruption  to  ongoing  tasks,  mission  execution,  or 
team  relationships.  Furthermore,  this  aviator  knows  that 
overloads  increase  the  risk  of  errors  and  poor  mission 
performance;  providing  support  to  overloaded  crew  members  is 
essential  to  effective  mission  execution. 

Overall,  testbed  participants'  crew  coordination  attitudes 
improved  significantly  as  a  result  of  the  crew  coordination 
training.  Commercial  airlines,  which  offer  their  own  versions  of 
crew  coordination  training  and  use  a  version  of  the  CMAQ  to 
evaluate  their  training,  define  success  as  a  positive  shift  in 
attitudes.  If  we  applied  the  commercial  airline  standard  to  this 
Army  program,  the  Army's  program  would  be  deemed  successful. 

Although  a  positive  shift  in  attitudes  is  desirable,  it  is 
not  sufficient  to  warrant  a  major  investment  in  training  and 
retraining  Army  aircrews.  For  this  reason,  DRC  was  asked  to 
evaluate  changes  in  measurable  behaviors,  crew-level  performance, 
and,  to  the  extent  possible,  the  impact  on  safety  of  the  crew 
coordination  training.  The  next  sections  report  on  those 
results . 

Basic  Quality  Ratings;  The  Aircrew  Coordination  Evaluation  (ACE) 
Checklist 


The  ACE  Checklist  is  a  behavioral  measure  that  was  developed 
by  ARIRWARU  and  DRC  (Simon,  1992) .  Based  on  the  results  of  an 
earlier  testbed  and  guidance  from  the  USAAVNC  Working  Group,  the 
ACE  Checklist  was  revised  (Grubb  et  al.,  1993)  for  use  in  the 
training  and  evaluation  materials  at  the  1992  Fort  Campbell 
testbed.  The  most  obvious  change  stemming  from  the  Working 
Group's  guidance  was  that  the  old  version  had  19  "dimensions,” 
whereas  the  new  version  has  13  "Basic  Qualities."  The  USAAVNC 
felt  that  the  Army  aviation  community  would  better  understand  the 
term  "basic  quality"  than  it  would  the  term  "dimension"  because 
"basic  quality"  is  used  during  initial  entry  rotary-wing  (lERW) 
training.  The  USAAVNC  Working  Group  also  felt  that  reducing  the 
number  of  Basic  Qualities  would  make  the  instrument  more 
manageable. 

The  USAAVNC  Working  Group  divided  crew  coordination 
behaviors  into  macro-  and  micro-level  behaviors.  Macro-level 
behaviors  are  evaluated  through  the  ACE  Checklist  and  are 
oriented  toward  hviman  relations  in  the  cockpit  and  how  crew 
members  perform  as  a  team.  Micro-level  behaviors  are  captured  by 
evaluating  performance  tied  to  specific  task  standards  associated 
with  individual  ATM  Tasks.  The  Working  Group  took  the  position 
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that  both  the  macro  and  micro  perspectives  must  be  applied  to 
effectively  understand  the  degree  to  which  crew  coordination  is 
implemented  in  the  cockpit. 

Each  Basic  Quality  was  designed  to  be  rated  by  an  IP- 
evaluator  on  a  seven-point  scale.  The  seven-point  scale  was 
anchored  at  the  1,  4,  and  7  levels  with  specific  behavioral 
descriptions  of  performance  at  those  levels.  IPs  were  instructed 
to  extrapolate  ratings  of  2,  3,  5,  and  6  from  the  descriptions 
given  at  the  1,  4,  and  7  levels  as  being  somewhat  better  or  worse 
than  the  anchored  description.  The  numbers  associated  with  the 
Basic  Quality  ratings  were  1  =  Very  Poor,  2  =  Poor,  3  =  Marginal, 
4  =  Acceptable,  5  =  Good,  6  =  Very  Good,  and  7  =  Superior.  IPs 
provided  the  Basic  Quality  ratings  on  a  grade  slip  designed  for 
the  testbed.  The  directions  for  using  the  ACE  Checklist  are 
provided  in  Appendix  B. 

Results  of  the  Basic  Quality  ratings.  Item-by-item  results 
in  the  form  of  frequency  tables  for  the  Basic  Qualities  composing 
the  ACE  Checklist  are  presented  in  Appendix  B.  Because  the 
Checklist  was  administered  twice,  once  during  the  pre-training 
evaluation  ride  (baseline)  and  a  second  time  during  the  post¬ 
training  evaluation  ride,  each  Basic  Quality  has  two  frequency 
tables  associated  with  it. 

Table  5  shows  a  comparison  of  Basic  Quality  item  means 
between  the  pre-  and  post-training  evaluations.  It  also  shows  a 
comparison  between  all  the  ACE  Checklist  item  means.  Cronbach's 
alpha  for  the  overall  scale  was  .95,  an  extraordinarily  high 
coefficient  that  attests  to  the  dependability  of  the  ratings 
derived  from  the  ACE  Checklist.  Ratings  were  compared  using  a 
paired  t-test  to  determine  whether  the  Basic  Qualities 
significantly  improved  between  the  pre-  and  post-training 
evaluation  missions. 

Table  5  shows  that  there  was  improvement  in  every  Basic 
Quality.  Despite  the  small  sample  size,  statistical  significance 
was  reached  on  12  of  the  13  Basic  Qualities.  On  average,  crews 
moved  from  a  rating  of  "marginal”  to  "acceptable"  within  the 
limited  timeframe  of  the  testbed.  The  goal  of  the  testbed 
training  was  to  achieve  a  rating  of  "acceptable."  Good  crew 
coordination  requires  practice,  therefore,  continued  training  is 
required  to  move  beyond  the  "acceptable"  level  of  performance. 

The  testbed  participants  verified  this  understanding  during  the 
testbed  debriefing  sessions;  they  expressed  a  desire  to  have  more 
training  time  to  achieve  better  ratings.  Once  the  aircrew 
coordination  program  is  fielded  and  continuation  training  becomes 
routine,  higher  levels  of  performance  should  be  readily 
achievable. 
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Table  5 


ACE  Checklist  Basic  Quality  Comparisons  between  Pre-  and  Post- 


training  Evaluation  Missions 


Basic  Quality 

Pre- 

training 

Post¬ 

training 

1 

Establish  and  maintain  flight  team 
leadership  and  crew  climate 

3.9 

ic-k 

5.1 

2 

Premission  planning  and  rehearsal 

3.1 

•kk 

4.6 

3 

Selection  of  appropriate  decision 
making  techniques 

3.2 

*k 

4.3 

4 

Prioritize  actions  and  distribute 
workload 

2.9 

kk 

4.4 

5 

Management  of  unexpected  events 

2.9 

kk 

4.1 

6 

Statements  and  directives  clear, 
timely,  relevant,  complete,  and 
verified 

3.1 

kk 

4.4 

7 

Maintenance  of  mission  situational 
awareness 

2.8 

k 

3.8 

8 

Decisions  and  actions  communicated 
and  acknowledged 

3.3 

kk 

4.6 

9 

Supporting  information  and  actions 
sought  from  crew 

3.3 

kk 

4.2 

10 

Crew  member  actions  mutually  cross 
monitored 

2.7 

kk 

3.9 

11 

Supporting  information  and  actions 
offered  by  crew 

3.2 

k 

4.2 

12 

Advocacy  and  assertion  practiced 

3.2 

kk 

4.1 

13 

Crew-level  after-action  reviews 
accomplished 

3.7 

4.2 

All 

13  Basic  Quality  Ratings 

3.2 

kk 

4.3 

*p  <  .05.  **p  <  .01. 


Although  it  is  pleasing  to  note  the  remarkable  success  the 
testbed  aviators  demonstrated  subsequent  to  the  training,  it  is 
noteworthy  that  Basic  Quality  13  was  not  well  evaluated.  The 
pre-training  scores  for  Basic  Quality  13  were  relatively  high  and 
little  movement  occurred  between  pre-  and  post-training.  The 
primary  reason  for  this  weakness  is  that  there  is  no  ATM  Task 
with  which  after-action  reviews  are  associated;  that  is,  there 
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are  no  ''micro-level”  behaviors  or  activities  to  guide  the 
process.  Thus,  it  is  difficult  for  IP  evaluators  to  adequately 
assess  performance  on  this  Basic  Quality.  This  problem  can  be 
rectified  by  (a)  including  after-action  reviews  in  the  ATM  for 
each  aircraft  and  (b)  improving  the  training  and  evaluation 
methods  for  after-action  reviews  in  the  field  exportable 
packages.  Based  on  this  lesson-learned  at  Fort  Campbell,  DRC  has 
improved  the  training  and  evaluation  materials  for  Basic  Quality 
13.  To  complement  this  improvement,  the  USAAVNC  should  take 
action  to  include  it  in  the  ATMs. 

ATM  Task  Ratings 

ATM  Tasks  contained  in  the  Final  Approved  Draft  of  the  newly 
revised  ATM  for  the  UH-60,  TC  1-212  (Department  of  the  Army,  July 
1992)  were  used  as  a  measure  of  behavior  and  performance  during 
the  validation  testbed.  Following  the  line  of  reasoning 
mentioned  previously,  the  ATM  Tasks  represent  the  micro-level 
behaviors  required  to  effectively  implement  aircrew  coordination. 
Many  of  the  ATM  Tasks  also  have  specific  perforaiance  criteria 
(standards)  which  must  be  met.  All  ATM  Tasks  include  both  a  crew 
coordination  and  a  technical  flight  skill  component. 
evaluators  rated  crews  on  various  ATM  Tasks  but  were  required  to 
rate  crews  on  18  selected  tasks  for  each  evaluation  mission.  The 
18  tasks  were  selected  for  the  following  reasons: 

•  They  provided  a  common  means  for  comparing  crews  on  pre- 
and  post-training  evaluation  missions. 

•  The  project  staff  felt  the  18  tasks  were  crew  coordination 
intensive. 

•  The  18  selected  tasks  were  represented  in  the  tactical 
scenarios  constructed  for  the  testbed. 

Three  forms  were  used  to  record  ATM  Task  and  Basic  Quality 
performance:  (a)  DA  Form  7172-R,  March  1992,  entitled  Battle- 

Rostered  Crew  Evaluation/Training  Grade  Slip  (Department  of  the 
Army,  July  1992) ;  (b)  the  Aircrew  Coordination  Training  Grade 
Slip  (Grubb  et  al.,  1993)  which  was  specifically  revised  for  the 
testbed  based  on  DA  Form  5700-R  entitled  Maneuver/ Procedure  Grade 
Slip  for  the  UH-60  RCM  (Department  of  the  Army,  July  1992);  and 
DA  Form  4507-2-R,  May  1987,  entitled  Continuation  Comment  Slip 
(Department  of  the  Army,  July  1992) .  For  purposes  of  this 
report,  these  three  forms  are  collectively  referred  to  as  the 
grade  slip.  Blank  grade  slips  are  provided  in  Appendix  C.  Note 
that  Basic  Quality  ratings  are  recorded  on  page  2  of  the  Aircrew 
Coordination  Training  Grade  Slip. 

The  grade  slip  used  for  the  testbed  required  space  for 
several  new  pieces  of  information: 
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•  The  IP-evaluators  needed  to  record  administrative 
information. 

•  The  IPs  were  asked  to  provide  an  overall  grade  for  the 
flight.  The  exact  criteria,  that  is  the  weighting  of  technical 
flight  skills,  crew  coordination,  and  mission  accomplishment, 
were  determined  by  the  IP. 

•  IPs  rated  ATM  Task  performance  on  a  four-point  scale:  S+, 
S,  S-,  and  U.^ 

•  When  a  crew  received  a  rating  of  S-  or  U  due  to  a  crew 
coordination  deficiency,  IPs  noted  two  Basic  Qualities 
contributing  to  the  problem.  IPs  were  encouraged  to  generously 
utilize  the  comment  section  for  use  during  the  aircrew  debriefs 
and  to  provide  notes  to  the  project  staff  about  other  concerns 
regarding  the  flight  or  the  rationale  for  providing  certain 
grades . 

Results  of  the  ATM  Task  grades.  Item-by-item  results  in  the 
form  of  frequency  tables  for  the  18  ATM  Tasks  and  the  overall 
grade  for  the  flight  are  presented  in  Appendix  C.  The  ATM  Task 
grades  were  translated  for  computer  analysis  so  that  S+  =  3,  S  = 
2,  S-  =  1,  and  U  =  0.  The  grade  slip  was  administered  twice: 
once  during  the  pre-training  evaluation  mission  (baseline)  and  a 
second  time  during  the  post-training  evaluation  mission; 
therefore,  each  ATM  Task  has  two  frequency  tables. 

Table  6  shows  a  comparison  between  each  of  the  18  ATM  Task 
grades  and  the  overall  grade  for  the  flight  on  the  pre-  and  post¬ 
training  evaluation  missions.  The  table  also  shows  a  comparison 
between  the  average  task  grade  for  all  18  ATM  Tasks.  Cronbach's 
alpha  for  the  ATM  Tasks,  taken  as  one  scale,  was  .87,  a  very  high 
reliability  coefficient.  Grades  were  compared  using  a  paired  t- 
test  to  determine  whether  ATM  Task  grades  significantly  improved 
between  the  pre-  and  post-training  evaluations. 

It  is  noteworthy  that  performance  for  every  ATM  Task 
improved.  Although  a  sample  this  small  is  unlikely  to  render 
statistically  significant  results,  nine  of  the  comparisons 
represent  significant  improvement  from  pre-  to  post-training 
scores.  Project  staff  noted  that  several  of  the  IPs  tended  to 
emphasize  the  seven  ATM  Tasks  for  which  there  was  statistically 
significant  improvement.  The  reason  the  IPs  tended  to  emphasize 
these  tasks  was  not  determined,  but  this  observation  and  the  fact 
that  the  post-training  scores  exceeded  the  pre-training  scores  by 
such  a  wide  margin,  indicates  that  were  IPs  to  provide 


^Typically,  unit  aviators  are  rated  on  a  two  point,  S  or  U, 
scale.  The  scale  used  in  the  testbed  was  the  same  as  is  used 
during  lERW  training. 
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Table  6 


ATM  Task  Grades  Comparisons  between  Pre-  and  Post-training 
Evaluations  _ 


UH-60  ATM  Task 

Pre- 

training 

Post¬ 

training 

1000 

Crew  Mission  Briefing 

.9 

-k-k 

2.1 

1004 

DA  Form  5701-R  (PPC) 

1.8 

2.1 

1007 

Before  Starting  Eng  through  A/C 
Shutdown 

1.7 

1.9 

1016 

Hover  Power  Check 

.3 

k* 

1.7 

1018 

VMC  Takeoff 

2.0 

2.2 

1023 

Fuel  Management  Procedures 

.3 

*k 

1.3 

1028 

VMC  Approach 

1.9 

2.0 

1068 

Emergency  Procedures 

1.2 

1.7 

1076 

Radio  Navigation 

1.8 

2.1 

1081 

Nonprecision  Approach 

1.2 

1.7 

1083 

Inadvertent  IMC/VHIRP 

1.8 

2.1 

1095 

A/C  Survivability  Equipment 

1.6 

k 

2.0 

2008 

Evasive  Maneuvers 

1.5 

1.8 

2009 

Multi  A/C  Operations 

1.6 

1.9 

2016 

External  Load  Operations 

1.4 

kk 

2.0 

2078 

Terrain  Flight  Mission  Planning 

1.3 

kk 

2.3 

2079 

Terrain  Flight  Navigation 

.9 

kk 

2.0 

2081 

Terrain  Flight 

1.8 

2.0 

Average  Score  for  the  18  Tasks 

1.4 

kk 

1.9 

Overall  Grade  for  Flight 

.8 

kk 

1.8 

*P  <  .05.  **E  <.01. 


concentrated  instruction  on  an  ATM  Task(s) ,  significant 
improvement  could  be  obtained.  This  conclusion  has  profound 
safety  implications  because  if  the  USASC  can  relay  problem  ATM 
Tasks  to  the  field,  IPs  are  clearly  capable  of  improving  ATM  Task 
performance,  thereby  enabling  the  breaking  of  safety  related 
error  chains  within  their  unit. 
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The  average  score  for  the  18  Tasks  improved  significantly. 
Before  the  training,  the  crews  were  able  to  obtain  an  S/S-  level 
of  ATM  Task  performance.  After  the  training,  the  aviators 
attained  an  S  level  of  performance.  As  a  group,  task-level 
performance  improved  by  one-half  a  grade  subsequent  to  the 
training.  The  overall  grade  for  flight,  a  grade  composed  of  an 
IP-determined  mix  of  technical  flying  skill,  crew  coordination, 
and  mission  effectiveness,  improved  by  a  full  grade.  That  is,  on 
the  pre-training  evaluation,  the  average  score  was  less  than  S-, 
whereas  after  the  training,  the  overall  grade  was  nearly  S. 

Results  of  the  ATM  Task  Grades  and  Related  Basic  Quality  Ratings 

When  an  ATM  Task  was  graded  S-  or  U  and  the  problem  involved 
crew  coordination,  IPs  noted  two  Basic  Qualities  contributing  to 
the  problem.  The  results  of  the  ATM  and  related  Basic  Quality 
analysis  are  shown  in  Table  7. 

Table  7  is  a  useful  diagnostic  tool  to  evaluate  the  specific 
strengths  and  weaknesses  of  a  unites  aircrews  and  of  a  crew 
coordination  training  program.  The  table  shows  which  ATM  Tasks 
were  problems  and  which  Basic  Qualities  were  absent,  thus 
contributing  to  ATM  Task  performance  problems.  There  were  marked 
improvements  in  the  following  Basic  Qualities: 

BQ  2  Premission  planning  and  rehearsal  accomplished 
BQ  3  Selection  of  appropriate  decision  techniques 
BQ  4  Prioritize  actions  and  distribute  workload 
BQ  5  Management  of  unexpected  events 

BQ  6  Statements  and  directives  clear,  timely,  relevant, 
complete,  and  verified 

BQ  7  Maintenance  of  mission  situational  awareness 
BQ  10  Crew  member  actions  mutually  cross-monitored 

Several  Basic  Qualities,  some  despite  their  improvement, 
need  to  receive  increased  emphasis  from  crew  coordination 
trainers  and  instructors.  These  include: 

BQ  7  Maintenance  of  mission  situational  awareness 

BQ  8  Decisions  and  actions  communicated  and  acknowledged 

BQ  9  Supporting  information  and  actions  sought  from  the  crew 

BQ  10  Crew  member  actions  mutually  cross  monitored 

BQ  11  Supporting  information  and  actions  offered  by  crew 

Basic  Qualities  8  and  9  showed  small  decrements  between  pre- 
and  post-training  evaluations.  Basic  Qualities  7  and  10  showed 
large  numerical  improvement  (-22  and  -31,  respectively)  but  were 
still  designated  by  IPs  as  areas  that  need  refinement.  Taken 
together,  Basic  Qualities  7,  8,  9,  10,  and  11  are  qualities  that 
may  suffer  from  battle-rostering. 
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Less  Than  Satisfactory  ATM  Task  Performance  and  Inhibiting  Basic  Qualities  for  Pre-  and 
Post-training  Evaluation* 
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♦Within  each  column,  numbers  to  the  left  are  pre-training  and  numbers  to  the  right  are  post-training. 

BQ1=Crew  climate  BQ2=Plan  rehearse  BQ3=Decision  techniques  BQ4=Work  load  BQ5=Unexpected  events 

BQ6=Positive  comm  BQ7=Situation  awareness  BQ8=Comm/acknowledged  BQ9=Info  sought  BQ10=Cross- monitor 

BQ11=Info  offered  BQ12=Advoc/assert  BQ13=After-action  Review  (AAR) 


Battle-rostering  breeds  familiarity.  Familiarity  can  be 
helpful  in  the  cockpit  because  crew  members  know  what  to  expect 
from  one  another;  they  are  likely  to  behave  in  a  manner  that  can 
be  anticipated  by  their  battle-rostered  fellow  crew  members,  and 
each  knows  the  other ^s  strengths  and  weaknesses.  On  the  other 
hand,  this  familiarity  may  also  breed  "assumption  creep" — pilots 
may  tend  to  assiame  the  following  when  battle-rostering 
familiarization  takes  place; 

1.  The  other  crew  member  is  maintaining  situational 
awareness  because  he  has  always  seemed  to  do  so  in  the  past. 
(Basic  Quality  7) 

2.  Decisions  and  actions  do  not  need  to  be  as  well 
articulated  and  verified  because  the  crew  has  had  similar 
situations  and  the  crew  is  performing  as  they  previously  did 
under  the  same  or  similar  circumstances.  (Basic  Quality  8) 

3.  Supporting  information  and  actions  do  not  need  to  be 
sought  because  the  pilot  flying  (PF)  assumes  that  the  pilot  not 
flying  (PNF)  or  other  crew  members  are  providing  all  the 
supporting  information  and  actions  the  PF  needs  to  maintain  safe 
and  efficient  flight.  (Basic  Quality  9) 

4.  Cross-monitoring  is  less  important  because  one  crew 
member  assumes  that  he  knows  what  other  crew  members  are  doing. 
Crew  members  automatically  assume  that  the  others  are  performing 
correctly  because  they  previously  performed  the  same  or  similar 
action  and  things  worked  out  well.  (Basic  Quality  10) 

5.  Supporting  information  and  actions  do  not  need  to  be 
offered  because  the  other  crew  members  assume  that  the  PF  is 
asking  for  all  the  required  information  and  actions  he  needs  to 
maintain  safe  and  efficient  flight.  (Basic  Quality  11) 

To  an  observer,  "assumption  creep"  may  seem  to  be 
complacency.  The  problem  cannot  be  verified  within  the  present 
data  set,  but  it  has  been  debated  within  the  USAAVNC  Working 
Group . 

Looking  across  the  rows  in  Table  7,  one  can  see  that 
certain  ATM  Tasks  continue  to  cause  problems  even  after  the  crew 
coordination  training.  Although  Table  6,  the  ATM  Task  Grades, 
also  shows  which  ATM  Tasks  were  problematic.  Table  7  presents  a 
more  informative  alternative  because  it  is  linked  to  the  Basic 
Qualities.  The  following  is  a  list  of  the  ATM  Tasks  that 
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continue  to  cause  problems  after  the  training  (i.e.,  those  tasks 
with  more  than  10  Basic  Quality  negative  notations: 

1016  -  Hover  Power  Check  (14  negatives) 

1023  -  Fuel  Management  Procedures  (11  negatives) 

1068  -  Emergency  Procedures  (16  negatives) 

1081  -  Nonprecision  Approach  (14  negatives) 

2008  -  Evasive  Maneuvers  (12  negatives) 

2079  -  Terrain  Flight  Navigation  (10  negatives) 

Collectively,  these  six  ATM  Tasks  accounted  for  77  (63%)  of  the 
negative  notations.  Given  this  type  of  information,  a  unit 
commander  would  know  which  ATM  Tasks  needed  to  be  emphasized  in 
his  crew  coordination  training  program.  As  previously  stated, 
when  IPs  emphasized  specific  ATM  Task  performance  during  the 
training,  it  was  observed  that  crews  markedly  improved  their 
performance  of  those  tasks. 

Mission  Performance  and  Efficiency 

Testbed  aircrews  were  given  a  utility  helicopter  tactical 
mission  to  perform  in  the  UH-60  flight  simulator.  Data  was 
collected  from  the  two  evaluation  missions  (pre-  and  post¬ 
training)  to  determine  whether  mission  performance  was  enhanced 
as  a  result  of  the  validation  testbed.  Important  components  of 
mission  performance  were  scrutinized,  but  the  central  question 
was  "What  is  the  military  worth  of  the  crew  coordination  training 
program? ” 

Two  scenarios  were  developed  for  the  testbed  evaluations. 
Project  staff  attempted  to  equalize  the  difficulty  of  the 
scenarios.  Assignment  of  crews  to  a  scenario  was  experimentally 
"counterbalanced”  so  that  there  was  a  50-50  split  for  each 
scenario  for  both  the  pre-  and  post-training  evaluation.  This 
experimental  design  technique  was  implemented  so  that  in  the  pre¬ 
training  evaluation,  eight  crews  were  assigned  scenario  one,  and 
eight  crews  were  assigned  scenario  two.  For  the  post-training 
evaluation,  those  crews  who  had  received  scenario  one  in  the  pre¬ 
training  evaluation  were  assigned  scenario  two,  and  those  who  had 
previously  received  scenario  two  were  assigned  scenario  one.  In 
this  way,  scenario  assignment  was  completely  counterbalanced  and 
results  were  viewed  by  looking  at  group  performance  for  the  pre- 
or  post-training  evaluations  without  any  effect  of  scenario 
assignment  on  group-level  performance. 
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A  diagram  of  scenario  two  is  presented  in  Figure  3  as  an 
example  of  the  two  scenarios.  The  mission  scenario  consisted  of 
five  phases: 

1.  Phase  1  (Planning)  —  Premission  Planning:  Crews  had 
1.5  hours  mission  planning  time  before  entering  the  simulator. 
This  phase  included  the  mission  briefing  and  all  preparatory 
tasks  associated  with  planning  a  tactical  mission.  The  tasks 
included  terrain  flight  mission  planning,  performance  planning, 
assigning  crew  member  responsibilities,  and  all  required 
briefings  and  brief-backs.  The  phase  ended  when  the  crew 
completed  all  of  the  briefings  and  entered  the  simulator. 

2.  Phase  2  (Positioning  and  Form  Up)  —  Movement  from  the 
Assembly  Area  (AA)  to  Pickup  Zone:  Starting  at  the  AA,  the  crew 
flew  to  pickup  zone  1  (PZl)  to  load  troops  for  the  cross-FLOT  air 
assault  mission.  Phase  2  included  an  aircraft  system 
malfunction,  which  should  have  resulted  in  a  precautionary 
landing  at  PZl.  The  phase  ended  when  the  crew  completed  the 
precautionary  landing.  While  loading  troops  at  PZl,  an  exact  was 
time  ("hard  time")  to  deliver  the  troops  at  landing  zone  1  (LZl) 
was  given  to  the  crew. 

3.  Phase  3  (Air  Assault)  —  Cross-FLOT  air  assault  from 
Pickup  Zone  1  to  Landing  Zone  1  to  Release  Point  2:  Acting  as 
lead  ship  in  a  flight  of  five  aircraft  with  no  changes  in  lead  or 
formation,  the  assault  involved  moving  troops  along  a  prescribed 
route  in  a  medium-to-high  threat  environment.  The  crew  entered 
corridor  number  1  by  crossing  start  point  1  (SPl)  -  (SPl  -  which 
is  where  timing  started  for  segment  one  of  the  mission) .  The 
crew  made  its  way  through  air  control  point  (ACP)  1,  2  and  3. 

Upon  arrival  at  release  point  1  (RPl) ,  the  crew  called  in  its 
arrival  to  a  Maneuver  Task  Force  Headquarters  to  determine 
whether  to  deliver  the  troops  to  the  primary  or  alternate  LZ.  In 
this  scenario,  crews  were  told  to  go  to  the  designated  alternate 
LZl.  After  delivering  the  troops  as  close  as  possible  to  the 
hard  time,  the  crew  flew  past  SP2  into  corridor  2,  past  ACP4  and 
to  RP2  where  timing  for  segment  1,  the  cross-FLOT  air  assault 
portion  of  the  mission,  ended.  The  phase  ended  at  PZ2.  The 
segment  also  included  a  minor  malfunction  (fuel  leak)  that  was 
administratively  removed  as  soon  as  the  crew  detected  and 
verbally  recognized  it. 
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15  Mission  Segments 


5  Phase  Tactical  Scenario 
•  Pre-Mission  Planning 


Figure  3.  Diagram  of  mission  scenario  two. 


4.  Phase  4  (Air  Movement)  —  An  external  load  air  movement 
from  Pickup  Zone  2  to  Landing  Zone  2 :  The  crew  executed  the  air 
movement  portion  of  the  mission  by  proceeding  to  pick  up  a  sling 
load  (an  M~102  howitzer  with  A-22  bag)  in  PZ2  to  resupply  a 
friendly  unit  located  near  the  PLOT.  After  picking  up  the  load, 
the  crew  flew  along  corridor  3  past  turn  point  (TP)  1  and  2  on 
its  way  to  the  planned  LZ  for  delivery.  The  crew  was  to 
accurately  navigate  within  the  prescribed  corridors  while 
avoiding  and  evading  threat. 

5.  Phase  5  (Return  to  Base)  —  Landing  Zone  2  to  Assembly 
Area:  After  delivering  the  howitzer,  the  crew  entered  corridor  4 
by  flying  back  the  way  they  had  come,  through  TP2  on  their  way  to 
AA,  which  was  the  planned  end  to  the  mission.  After  reaching 
TP2,  the  crew  encountered  inadvertent  instrument  meteorological 
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conditions  (IMC) ,  performed  vertical  helicopter  instrument 
recovery  procedures  (VHIRP) ,  and  executed  a  nondirectional  beacon 
(NDB)  approach  to  a  recovery  airfield.  The  crew  planned  and 
executed  the  nonprecision  approach  to  transition  back  to  visual 
meteorological  conditions  (VMC) .  The  phase  ended  when  the  crew 
completed  a  safe  landing. 

During  the  mission,  the  crews  encountered  threat  in  the  form 
of  anti-aircraft  guns  (ZSU  23-4)  and  missile  sites  (SA7/8) .  For 
the  most  part,  threat  was  minimal  if  crews  stayed  within  the 
assigned  corridors.  When  crews  encountered  threat  warnings,  they 
had  to  take  evasive  action  or  risk  being  engaged. 

Two  emergencies,  one  minor  malfunction  and  one  major 
emergency,  were  programmed  into  the  scenario.  The  minor 
malfunction  was  a  slow  fuel  leak.  As  mentioned  above,  as  soon  as 
a  crew  member  announced  awareness  of  the  leak,  the  simulator 
instructor  operator  stopped  it.  The  major  emergency  was  a 
decreasing  rotor  RPM  that  would  force  a  landing  "as  soon  as 
practicable."  The  emergency  occurred  in  a  place  where  the 
optimal  landing  area  was  PZl  and  was  administratively  corrected 
once  on  the  ground.  No  negative  mission  impact  was  experienced. 

Mission  performance  data  collection.  Project  staff  measured 
mission  performance  using  a  variety  of  data  collection 
techniques.  Each  simulator  mission  was  recorded  using  four  video 
cameras  multiplexed  onto  one  video  picture.  All  intercom  and 
radio  communications  and  aural  warnings  were  recorded  onto  the 
videotape.  Two  cameras  were  situated  in  the  cockpit  aimed  at 
each  of  the  aviators.  A  third  camera  was  placed  in  the  simulator 
computer  room  and  aimed  at  a  specially  programmed  display  to 
capture  critical  flight  parameters:  altitude,  airspeed,  and 
heading.  The  fourth  camera,  also  in  the  simulator  computer  room, 
was  aimed  at  a  high  resolution  monitor  that  displayed  the  right 
seat's  forward  windscreen  out-the-window  view.  Figure  4  shows 
what  was  seen  on  the  video  display  monitors  and  recorded  for 
later  review. 

During  each  simulator  mission,  two  members  of  the  project 
staff  used  a  small  computer  and  data  entry  software  to  create 
computer-generated  log  files  of  mission  critical  events  in  near- 
real  time.  The  computer  clock  was  software  controlled  to 
synchronize  log  file  entries  with  the  videotape  clock  so  that 
both  the  videotape  and  the  log  file  recorded  mission  times  from 
time  zero.  The  log  file  contained  such  things  as  times  of 
departure  and  arrival  at  various  points,  threat  encounter 
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Figure  4.  The  data  collection  video  scene:  Four  cameras 
combined  onto  one  video  display. 

duration  and  outcome,  timing  and  ratings  of  emergency  procedures, 
instrument  recovery  flight  parameters  and  crew  interactions, 
whether  there  was  an  occurrence  of  a  crew-caused  mission 
threatening  error,  how  much  of  the  mission  was  accomplished  by 
the  crew,  and  various,  ad  hoc  comments  by  the  observers. 

Simulator-generated  plots  of  flight  profiles  were  used  to 
reconstruct  the  exact  flight  path  of  the  aircraft.  The  plots 
also  showed  which  enemy  threat  systems  were  activated  throughout 
the  mission. 

IP-evaluators  kept  track  of  the  mission  using  evaluator 
worksheets.  The  evaluator  worksheets  contained  an  outline  of  the 
mission,  mission  segment  descriptions,  and  ATM  Tasks  associated 
with  each  segment .  The  IPs  used  the  worksheets  to  record  crew 
performance  regarding  the  ATM  Tasks  and  the  Basic  Qualities.  IPs 
also  recorded  how  well  crews  performed  required  emergency  steps, 
when  the  minor  malfunction  was  detected,  and  heading  and  altitude 
deviations  on  the  IMC  approach.  (Note:  Where  possible,  such  as 


for  the  IMG  approach,  project  staff  confiirmed  IP  observations 
through  post-testbed  review  of  the  videotapes . ) 

Mission  performance  -  Data  analysis.  Mission  performance 
data  were  analyzed  according  to  the  following  categories: 
navigation,  threat,  emergencies  and  malfunctions,  instrviment 
recovery,  and  overall  mission  performance.  Some  of  the  analysis 
required  extensive  review  of  the  videotapes.  On  occasion, 
videotaped  events  were  compared  with  the  log  files  and  the 
simulator  plots. 

A  short  digression  at  this  point  is  warranted:  The  data 
reported  in  the  next  several  sections  is  quantitative  in  nature 
and  the  numbers  reflect  the  performance  enhancements  and  the 
military  benefits  engendered  through  the  crew  coordination 
training  conducted  at  Fort  Campbell.  The  numbers,  however,  do 
not  tell  the  whole  story.  Project  staff  and  several  members  of 
the  USAAVNC  Working  Group  have  viewed  the  mission  videotapes. 
Without  exception,  some  of  the  most  impressive  aspects  of  the 
differences  in  pre-  and  post-training  conditions  were  qualitative 
in  nature.  Post-training  crews  work  better  as  teams,  they  plan 
better  during  the  premission  activities  and  during  the  flight, 
they  exchange  information  better,  manage  workload  more 
efficiently — taking  advantage  of  one  another's  talents  and 
skills,  and  they  check  each  others'  performance  as  the  mission 
unfolds.  These  crews  were  "ahead  of  the  aircraft"  nearly  all  of 
the  time.  They  were  in  control.  Furthermore,  when  events  and 
the  aircraft  became  troublesome,  they  handled  problems 
efficiently  and  with  mission  objectives  in  mind.  The  qualitative 
change  in  the  cockpit  was  striking. 

In  the  following  sections,  summary  tables  are  presented  for 
each  performance  measurement  area.  In  most  cases,  the  tables 
consist  of  four  columns:  The  first  column  is  the  variable  under 
consideration;  the  second  and  third  columns  are  the  pre-  and 
post-training  results,  respectively;  and  the  fourth  column 
presents  a  summary  statistic  showing  the  change  from  pre-  to 
post-training.  For  nearly  every  performance  variable,  the  change 
is  in  the  expected  direction;  there  is  improvement  after  the 
training.  However,  less  is  better  for  some  of  the  variables. 

For  example,  it  is  better  to  have  fewer  course  deviations,  fewer 
crashes ,  etc . 

Mission  performance  -  Navigation.  Results  of  the  analysis 
of  navigation  are  presented  in  Table  8  and  discussed  in  the 
following  paragraphs. 
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Table  8 


Pre-  and  Post-training  Evaluation  of  Navigation  Performance  Using 
a  t-Test  for  Correlated  Samples 


Naviaation  measure 

Pre- 

trainina 

Post- 

trainina 

Number  of  course  deviations  greater 
than  500  meters  beyond  standard 

47  * 

26 

Percent  of  time  off  course 

44%  ** 

21% 

Number  of  crews  arriving  at  correct 
landing  zone 

10  * 

16 

Average  deviation  time  for  crews 
arriving  at  the  correct  landing  zone 

4.5 

minutes 

3.0 

minutes 

Number  of  crews  arriving  within  1 
minute  of  hard  time  at  landing  zone 

3 

7 

*E  <  .05.  **E  <  .01 


Number  of  course  deviations  greater  than  500  meters  beyond 
standard:  The  four  planned  corridors  were  one  kilometer  wide.  A 
deviation  was  counted  when  crews  strayed  outside  the  corridor 
more  than  500  meters.  On  occasion,  crews  purposely  left  the 
corridor  to  avoid  a  threat  system.  These  avoidance  maneuvers 
were  not  counted  against  the  crew  if  they  (a)  knew  they  were 
making  an  excursion  and  (b)  were  able  to  reorient  onto  the 
planned  course  after  the  threat  subsided.  Crews  deviated  from 
the  corridor  45%  less  in  the  post-training  evaluation  than  they 
did  in  the  pre-training  evaluation. 

Percent  of  time  off  course:  Once  a  course  deviation  (as 
defined  in  the  previous  paragraph)  occurred,  crews  attempted  to 
reorient  the  aircraft  back  onto  the  course.  Pre-training  crews 
spent  44%  of  their  time  misoriented,  whereas  post-training  crews 
spent  only  21%  being  misoriented,  an  improvement  of  23%. 

Number  of  crews  arriving  at  the  correct  landing  zone:  Crews 
were  told  to  deliver  troops  to  a  specific  LZ.  Only  10  of  16  pre¬ 
training  crews  arrived  at  the  correct  LZ,  whereas  16  of  16  crews 
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arrived  at  the  correct  LZ  in  the  post-training  evaluation.  This 
represents  a  38%  improvement  in  being  able  to  deliver  troops  to 
the  correct  place. 

Average  deviation  time  for  crews  arriving  at  the  correct 
landing  zone:  Crews  were  given  a  "hard  time”  to  arrive  at  the 
LZ.  This  measure  shows,  on  average,  how  close  crews  came  to 
meeting  the  hard  time.  More  precision  is  better  than  less. 

During  the  pre-training  evaluation,  crews  deviated  (+  or  -)  an 
average  of  4.5  minutes  from  the  hard  time;  during  the  post¬ 
training  evaluation,  crews  deviated  an  average  of  3.0  minutes, 
representing  a  33%  improvement. 

Number  of  crews  arriving  within  1  minute  of  hard  time  at 
landing  zone:  Because  no  Army-wide  standard  exists  for  meeting  a 
hard  time,  allowable  deviations  are  locally  and  situationally 
determined.  The  reasons  for  this  precision  vary,  but  in  the 
testbed  mission  when  crews  were  performing  a  cross-FLOT  air 
assault,  accompanying  attack,  helicopters  were  scheduled  to  place 
preparatory  fires  on  the  LZ  1  minute  prior  to  the  hard  LZ  time. 

In  the  second  scenario,  a  second  lift  element  was  due  to  arrive  3 
minutes  after  the  first  lift.  All  of  these  operational  factors 
made  timing  critical  and  the  aircrews  were  briefed  on  the 
importance  of  accurately  meeting  the  hard  time.  Only  three  pre¬ 
training  crews  arrived  within  the  1  minute  window,  whereas  seven 
post-training  crews  (nearly  50%  of  the  crews)  arrived  within  the 
1  minute  window.  This  represents  a  25%  improvement  from  pre-  to 
post-training  evaluations . 

Mission  performance  -  Threat.  The  simulator  scenarios  were 
constructed  with  antiaircraft  gun  and  missile  threat  systems 
activated.  The  location  of  the  threat  sites  were  chosen  so  that, 
for  the  most  part,  when  crews  stayed  within  the  assigned 
corridors  and  altitude,  few  or  no  threat  systems  were 
encountered.  Project  staff,  acting  as  observers,  recorded  each 
threat  system  encounter  in  the  computerized  logger  file.  Each 
encounter  was  logged  and  time-stamped.  Subsequent  to  the 
testbed,  every  threat  encounter  was  reviewed  using  the  mission 
videotapes . 

Three  considerations  guided  the  analysis  of  the  videotapes. 
First  was  the  warning  state.  There  are  three  warning  states  of 
the  APR-39  in  the  UH-60  cockpit:  search,  track,  and  weapons 
activity.  The  analysis  counted  only  track  and  weapons  activity 
as  a  valid  warning  or  encounter.  Second  was  the  length  of  the 
warning.  Although  there  is  no  Army-wide  standard  regarding  an 
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acceptable  length  of  a  threat  encounter,  members  of  the  USAAVNC 
Working  Group  and  Fort  Campbell  IPs  informed  the  project  staff 
that  10  seconds  was  a  good  rule-of-thumb  for  acceptability — more 
than  10  seconds  presents  extreme  danger  to  the  crew.  Thus, 
encounters  greater  than  10  seconds  were  tabulated.  The  third 
consideration  for  each  threat  encounter  was  the  outcome.  There 
were  two  outcomes  in  the  analysis.  The  good  or  acceptable 
outcome  was  "broke  lock."  A  detrimental  outcome  was  defined  as 
either  became  lost  or  misoriented,  crashed,  or  took  hits.  Based 
on  these  three  considerations  (number  of  warnings,  duration  of 
warning,  and  outcome) ,  four  rating  criteria  were  established  for 
the  testbed  aircrews.  These  criteria  are  presented  in  Table  9. 


Table  9 

Rating  Criteria  Used  for  Threat  Encounters 


Rating 

#  of  warnings 

#  of  warnings 
>  10  seconds 

#  of 

detrimental 

outcomes 

3  =  Superior 

4  or  fewer 

0 

0 

2  =  Good 

8  or  fewer 

1  or  fewer 

0 

1  =  Marginal 

12  or  fewer 

2  or  fewer 

1  or  fewer 

0  =  Very  Poor 

More  than  12 

More  than  2 

More  than  1 

The  four  ratings  shown  in  Table  9  were  used  so  that  a  crew 
had  to  meet  all  three  criteria  to  make  a  rating  (i.e.,  an  "and" 
Boolean  operator  was  used) .  For  example,  a  crew  with  four  or 
fewer  warnings,  one  of  which  was  greater  than  10  seconds,  and 
with  no  detrimental  outcomes,  would  be  given  a  rating  of  "good". 

A  crew  with  eight  or  fewer  warnings,  none  greater  than  10 
seconds,  that  became  misoriented  as'  a  result  of  taking  evasive 
maneuvers,  would  be  given  a  rating  of  "marginal." 

Results  of  the  analysis  of  threat  encounters  are  presented 
in  Table  10.  During  the  post-testbed  analysis,  it  became  evident 
that  the  threat  was  not  working  properly  for  three  of  the  crews 
during  the  pre-training  evaluation.  Therefore,  only  13  crews 
were  rated  during  the  pre-training  evaluation,  whereas  16  crews 
were  rated  on  the  post-training  evaluation. 
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Table  10 


Pre-  and  Post-training  Evaluation  of  Threat  Encounters 


Pre- 

Post- 

Measure 

training 

training 

Crews  receiving  a  "superior" 

7  of  13 

11  of  16 

or  "good"  rating 

(54%) 

(69%) 

As  shown  in  Table  10,  the  improvement  from  pre-  to  post¬ 
training  ratings  was  15%.  Other  positive  changes  from  pre-  to 
post-training  evaluations  were  also  recorded.  For  example,  there 
were  only  four  pre-training  "superior"  ratings,  but  there  were 
six  post-training  "superior"  ratings;  and  although  there  were 
four  "very  poor"  ratings  on  the  pre-training  evaluation,  there 
were  only  two  such  ratings  on  the  post-training  evaluation.  Of 
the  six  crews  receiving  a  "marginal"  or  "very  poor"  rating  on  the 
pre-training  evaluation,  five  crews  improved.  With  these 
additional  pieces  of  information  in  mind,  it  is  safe  to  conclude 
that  the  15%  improvement  in  dealing  with  threat  is  a  conservative 
estimate  of  the  improvement  as  a  result  of  the  testbed  training 
activities . 

Mission  performance  -  Emergencies.  During  the  missions, 
crews  were  given  two  types  of  unexpected  aircraft  system 
malfunctions:  One  was  minor  (a  slow  fuel  leak  that  was  stopped 

as  soon  as  a  crew  member  announced  the  problem) ,  and  the  other 
was  a  major  emergency  that  entailed  either  an  increasing  or 
decreasing  rotor  RPM.  The  major  emergency  required  the  crew  to 
work  in  concert  to  identify  and  isolate  the  problem  and  to 
successfully  complete  a  landing  as  soon  as  practical.  Table  11 
presents  the  outcomes  of  these  measures. 

Number  of  crews  detecting  minor  malfunction:  This  measure 
is  an  excellent  assessment  of  the  crews'  situational  awareness 
and  ability  to  effectively  manage  workload.  Crews  did  far  better 
on  this  measure  subsequent  to  the  crew  coordination  training. 

Number  of  crews  correctly  diagnosing  major  emergency:  There 
was  a  minor  decrement  in  performance  from  pre-  to  post-training 
evaluations  on  this  measure.  The  technical  nature  of  the  major 
emergency  presented  the  crew  with  an  uncertain  situation. 

Because  the  aircraft  gross  weight  was  light  and  airspeed  was  low, 
there  was  a  low  power  demand;  thus,  the  emergency  did  not  always 
cause  continuing  loss  or  gain  in  rotor  RPM  (decreasing  or 
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Table  11 


Results  of  Pre-  and  Post-training  Evaluation  of  Aircraft 
Emergencies _ 


Pre- 

Post- 

Measure 

training 

training 

Number  of  crews  detecting 
minor  malfunction 

10  ** 

16 

Number  of  crews  correctly 
diagnosing  major  emergency 

12 

11 

Number  of  crews  with  a  safe 

16 

16 

landing  subsequent  to  the 
aircraft  emergency 

**E  <  .01 


increasing  %  rotor  RPM) .  In  those  cases  where  RPM  stabilized 
within  acceptable  parameters,  crews  often  believed  that  they  had 
experienced  a  different  emergency;  usually  they  thought  a  torque 
split  had  occurred.  As  a  result,  crews  did  not  always  properly 
diagnose  the  emergency  or  complete  the  emergency  procedure,  which 
was  to  place  the  affected  engine's  power  lever  to  environmental 
control  unit  (ECU)  lockout  position.  Qualitative  data,  based  on 
videotape  review,  shows  improvement  in  crew  interactions  while 
coping  with  the  major  emergency  and  confirms  the  modest 
improvement  in  emergency  procedure  ATM  Task  performance. 

Number  of  crews  with  a  safe  landing  subsequent  to  the 
aircraft  emergency:  All  crews  remained  unscathed  subsequent  to 
the  aircraft  emergency  in  both  the  pre-  and  post-training 
evaluations.  Some  investigation  was  required  to  explain  this 
measure  and  the  one  above;  number  of  crews  correctly  diagnosing 
the  major  emergency.  It  was  expected  that  crews  would  improve  on 
these  two  measures,  but  they  did  not.  One  reason  for  this  may  be 
that  emergency  procedures  are  often  practiced  in  the  simulator; 
thus,  aircrews  expect  emergencies  in  the  simulator  environment 
because  they  routinely  practice  emergency  procedures  there. 
Another  reason  may  be  that  the  emergency  presented  in  the 
scenario  was  relatively  benign;  that  is,  there  was  little  danger 
of  crashing  and  losing  the  aircraft.  If  a  more  serious  emergency 
had  been  scripted  into  the  scenario,  such  as  loss  of  hydraulics, 
the  outcome  might  have  been  different. 
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The  lack  of  results  on  the  above  two  measures  is  not 
critical.  The  primary  reasons  for  scripting  emergencies  into  the 
scenarios  were  (1)  to  present  aircrews  with  stressful  situations 
in  which  crew  coordination  would  be  useful  and  (2)  to  provide  IP- 
evaluators  with  ample  opportunity  to  assess  how  well  aircrews 
practiced  crew  coordination.  These  two  objectives  were  achieved. 

Mission  performance  -  Instrument  recovery.  Instrument 
recovery  is  a  crew  coordination-intensive  activity.  Crews  were 
measured  on  their  performance  of  an  entry  into  inadvertent  IMC 
whereupon  they  had  to  execute  VHIRP  and  a  nonprecision  instrument 
approach  using  a  nondirectional  beacon  as  the  only  external 
navigation  aid.  Several  objective  measures  of  their  performance 
were  taken,  and  one  rating  scale  was  applied.  The  rating  scale, 
applied  by  project  staff  in  their  role  as  observers  and  later 
verified  by  reviewing  the  videotapes  after  the  testbed,  is 
presented  in  Table  12. 


Table  12 

Rating  Criteria  Used  for  Instrument  Approach  Planning  and 
Execution 


Rating 

Description 

3  =  Superior 

Both  crew  members  review,  discuss,  and 
rehearse  the  approach;  PNF  assists  the  PF 
throughout  the  approach. 

2  =  Good 

One  crew  member  reviews  the  approach,  briefs 
the  other  prior  to  executing  the  approach, 
then  talks  him  through  it. 

1  =  Marginal 

One  crew  member  reviews  the  approach  and  talks 
the  other  through  it. 

0  =  Very  Poor 

One  crew  member  reviews  and  executes  the 
approach  with  no  assistance  from  the  other. 

The  results  of  measuring  instrument  recovery  performance  are 
presented  in  Table  13. 

Number  of  crews  with  all  five  recovery  procedure  steps 
correct:  Upon  inadvertent  entry  into  IMC  conditions,  crews  had  to 
execute  five  VHIRP  recovery  steps  to  transition  to  instrument 
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Table  13 


Results  of  Pre-  and  Post-training  Evaluation  of  the  Instrument 
Recovery _ _ 


Instrument  recovery  measure 

Pre¬ 

training 

Post¬ 

training 

Number  of  crews  with  all  5 
recovery  procedure  steps  correct 

9 

13 

Number  of  crews  with  successful 
transition  from  VMC  to  IMC 

14 

16 

Rating  of  approach  planning® 

1.6 

1.9 

Number  of  crews  with  "superior” 
approach  planning 

2 

5 

Number  of  crews  with  no  altitude 
deviations  greater  than  100  ft 
beyond  standard 

g  * 

14 

Number  of  crews  not  descending 
below  missed  approach  point  (MAP) 
early 

10  * 

15 

Number  of  crews  with  properly 
timed  inbound  leg 

11 

14 

^Rating  based  on  the  scale  presented  in  Table  12. 
*E  <  .05 


flight.  IPs  observed  the  number  of  correct  steps  taken  by  each 
crew.  In  the  pre-training  ride,  9  of  16  crews  (56%)  performed 
all  five  steps  correctly;  during  the  post-training  evaluation,  13 
of  16  crews  (81%)  performed  correctly,  representing  a  25% 
improvement  on  this  measure. 

Number  of  crews  with  successful  transition  from  VMC  to  IMC: 
Two  crews  did  not  safely  transition  from  VMC  to  IMC.  They 
crashed.  Although  only  a  small  percentage  of  aircrews  improved 
their  performance  from  pre-  to  post-training,  this  is  an 
important  measure  because  it  is  life  and  materiel  threatening. 
This  leads  to  an  interesting  point:  Even  relatively  small 
improvements  may  have  real-world  implications  much  greater  than 
would  first  appear. 
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Rating  of  approach  planning ,  and  number  of  crews  with 
superior  approach  planning:  Crews  were  rated  on  these  two 
related  measures  as  having  performed  substantially  better  after 
the  crew  coordination  training. 

The  last  three  items  in  Table  13,  altitude  deviations 
greater  than  100  feet  beyond  standard;  not  descending  below  MDA 
early;  and  the  timing  of  the  inbound  leg,  all  measured  how  well 
crews  executed  the  instrument  approach.  Other  measures  were 
considered  for  use  here,  such  as  heading  deviations,  but  were  not 
included  because  they  showed  little  difference  between  pre-  and 
post-training  performance.  They  were  also  considered  to  be  less 
important  than  measures  such  as  altitude  deviations.  There  was 
substantial  improvement  for  each  of  these  measures. 

Collectively,  the  seven  measures  listed  in  Table  13  clearly 
show  that  the  activities  associated  with  inadvertent  entry  into 
IMC  and  instrument  recovery  improved  markedly  by  the  time  the 
crews  were  evaluated  during  the  post-training  ride.  Because 
successful  performance  of  these  activities  is  highly  dependent  on 
crew  coordination,  and  performance  improved  after  training,  it 
appears  that  the  training  improved  the  crews'  performance. 

Mission  performance  -  Overall  measures.  Several  global 
measures  of  mission  performance  dealing  with  overall  mission 
accomplishment  were  used.  Whereas  most  of  these  measures  were 
objective  measures  of  performance,  one  measure  was  subjective: 
"mission  threatening  error."  This  measure  was  created  to  capture 
unexpected  events  that  negatively  affect  mission  accomplishment. 
For  instance,  during  nap-of-the-earth  flight,  a  crew  hits  the 
tail  wheel  on  the  ground,  does  not  crash,  and  continues  the 
mission  with  only  limited  damage  to  the  aircraft.  This  type  of 
event  would  not  be  reflected  in  the  other  performance  measures; 
therefore,  a  scale  was  constructed  to  capture  this  type  of  event. 
The  project  staff  mission  observers  used  this  scale  and  later 
verified  it  by  reviewing  the  videotapes.  The  mission  threatening 
error  scale  is  presented  in  Table  14. 
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Table  14 


Rating  Criteria  Used  for  Mission  Threatening  Error  Scale 


Rating 


Description 


4  =  None 


No  occurrence  of  mission  threatening 
error.® 


3  =  Satisfactory  + 
S+ 


2  =  Satisfactory 
S 


Without  any  damage  to  the  aircraft, 
the  crew  completes  both  the  cross-FLOT 
air  assault  and  external  load  missions 
within  the  time  and  location  accuracy 
required  by  the  scenario. 

Minor  damage  due  to  terrain  strike 
(e.g.,  tail  wheel  contact)  or  enemy 
fire  (e.g. ,  stabilator  hits)  or  the 
crew  misses  the  LZ  hard  time  by  one  to 
three  minutes  or  releases  the  external 
load  more  than  500  meters  from  the 
desired  location  and/or  misses 
navigation  check  points.  However, 
this  crew  reasonably  completes  all 
mission  objectives. 


1  =  Satisfactory  -  Aircraft  damage  (e.g.,  blade  strike, 

S-  enemy  fire)  occurs  that  would  have 

interrupted  the  mission  for  repairs  or 
the  mission  is  abbreviated  due  to 
navigation  errors.  The  crew  manages 
to  recover  to  complete  only  part  of 
the  mission  (e.g.,  deliver  troops  more 
than  three  minutes  past  the  LZ  hard 
time,  substitute  an  internal  load  for 
the  planned  external  load) . 


0  =  Unsatisfactory  Obstacle  strike,  terrain  strike,  or 

U  enemy  fire  destroys  the  aircraft  or 

crew  errors  (e.g.,  repeated 
misorientations)  culminate  in 
situations  where  the  crew  is  unable  to 
recover  to  deliver  the  troops  or 
external  load  within  any  reasonable 

_ time  or  approximate  location. _ 

®No  crews  received  a  rating  of  4.  This  was  expected  given  the 
complex,  battle-oriented  mission  that  the  crews  were  flying. 
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To  summarize  the  rating  factors  in  Table  14: 


Satisfactory+  (S+) 
Satisfactory  (S) 
Satisfactory-  (S-) 

Unsatisfactory  (U) 


Everything  is  fine;  all  objectives  met 

Minor  problems;  objectives  nearly  met 

Recoverable  problems  or  damage; 
objectives  partially  complete 

Irrecoverable  problems  or  damage; 
objectives  incomplete 


Table  15  presents  the  results  of  the  Mission  Overall 
measures . 


Table  15 

Results  of  Pre-  and  Post-training  Evaluation  for  Overall  Measures 


Overall  measure 

Pre¬ 

training 

Post¬ 

training 

Mission  threatening  error® 

1.1  * 

1.9 

Number  of  crews  receiving  ”3+"  or 
••S”  mission  error  ratings 

6  * 

12 

Percent  of  mission  segments 
accomplished 

66%  ** 

86% 

Number  of  crews  completing  cross- 
FLOT  and  external  load  missions 

9  * 

16 

Number  of  aircraft  crashes^ 

7 

4 

^Rating  based  on  the  scale  presented  in  Table  14. 
*^Some  aircrews  had  more  than  one  crash. 

*P  <  .05.  **p  <  .01 


Table  15  presents  the  most  compelling  evidence  of  the 
military  benefits  of  the  crew  coordination  training  provided  to 
the  Fort  Campbell  testbed  aircrews.  The  measures  focus  on  how 
well  the  crews  accomplished  the  mission  and  how  safely  they 
performed. 
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Mission  threatening  error:  The  testbed  aircrews  averaged 
”S-"  performance  during  the  pre-training  evaluation  and  "S” 
performance  on  the  post-training  evaluation.  In  terms  of 
improvement  on  the  rating  scale  (movement  of  . 8  on  the  3 . 0 
scale),  performance  improved  27%. 

Number  of  crews  receiving  "S"  or  "S+"  error  ratings:  This 
variable  is  another  way  of  looking  at  mission  threatening  error. 
In  the  case  of  the  testbed  aircrews,  six  additional  crews  (more 
than  one-third  of  the  crews)  improved  their  performance  so  that 
they  received  an  "S”  or  ”8+"  rating. 

Percent  of  mission  segments  accomplished:  This  measure  was 
determined  after  the  testbed  by  reviewing  the  observer  logs,  the 
simulator  plots,  and  the  videotapes.  Each  of  the  scenarios  was 
divided  into  segments.  There  were  15  segments  for  each 
evaluation  mission.  The  15  segments  included  reaching  the  air 
control  points,  report  points,  landing  zones,  etc.  (Refer  to 
Figure  3  to  see  the  segments  for  scenario  two.)  During  the  pre¬ 
training  evaluation  missions,  crews  averaged  66%  mission 
accomplishment;  on  the  post-training  evaluation,  crews  averaged 
86%  mission  accomplishment. 

Number  of  crews  completing  cross-FLOT  and  external  load 
missions :  This  measure,  derived  the  same  way  as  "the  percent  of 

mission  segments  accomplished”  (see  preceding  paragraph) ,  is  a 
"bottom  line"  number.  The  two  central  military  objectives  of  the 
mission  were  to  perform  an  air  assault  and  an  air  movement. 
Despite  any  other  intervening  difficulties,  the  crews  had  to 
attain  these  two  objectives.  In  the  pre-training  evaluation, 
only  66%  of  the  crews  accomplished  the  two  objectives.  In  the 
post-training  evaluation,  100%  of  the  crews  accomplished  the 
mission.  During  the  training,  crews  learned  techniques  to  keep 
focused  on  what  is  important  and  to  deal  with  emergencies  and 
unforeseen  problems  in  such  a  way  that  they  do  not  lose  sight  of 
mission  objectives.  Field  commanders  should  be  aware  of  the 
statistic  for  this  measure  because  it  is  the  one  that  most 
clearly  shows  the  impact  of  crew  coordination-trained  aircrews  on 
military  effectiveness. 

Number  of  aircraft  crashes:  One  of  the  goals  of  crew 
coordination  training  is  to  enhance  the  margin  of  flight  safety. 
The  data  presented  in  Table  15  support  the  idea  that  safety  is 
enhanced  subsequent  to  the  training.  Although  the  results  are 
impressive,  one  is  cautioned  against  making  strong  conclusions 
based  on  this  one  data  set.  First,  two  crews  had  more  than  one 
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crash  in  the  pre-training  evaluation.  Second,  aviators  may  be 
less  cautious  in  the  simulator  because  a  crash  in  the  simulator 
does  not  cause  any  real  harm.  Third,  the  simulator  visuals  do 
not  present  the  rich  textural  and  topographic  clues  present  in 
the  real  world.  Fourth,  the  testbed  sample  is  too  small  to  allow 
for  a  definite  conclusion  that  43%  of  aircraft  crashes  will  be 
avoided  after  the  crew  coordination  training.  Nevertheless,  this 
measure  is  important  as  a  safety  indicator.  At  a  minimum,  it 
shows  that  crew-level  situation  awareness  is  markedly  enhanced 
after  the  crew  coordination  training.  Furthermore,  it  may  point 
to  a  potential  dramatic  reduction  in  accidents. 

Summary  of  findings  from  aircrew  and  instructor  debriefings. 
Crew  members  and  instructors  were  debriefed  during  the  last  two 
days  of  the  testbed.  Prior  to  the  debriefs,  participants  were 
given  the  debrief  interview  questions  to  make  notes  on  the  items 
they  wanted  to  discuss  during  the  debriefs.  Many  of  the 
participants  wrote  answers  to  the  interview  questions  and  gave 
them  to  the  project  staff. 

Four  groups  of  eight  aviators  were  interviewed  during  the 
first  day  of  the  two-day  debriefing  period.  Each  group  debrief 
took  two  hours.  The  project  staff  recorded  each  group's 
responses  and  discussions.  On  the  morning  of  the  second  day, 
instructors  were  interviewed  in  a  two-hour,  one-on-one  situation. 
In  the  afternoon,  instructors  participated  in  a  two-hour  group 
debrief.  As  in  the  aviator  debriefs,  project  staff  recorded 
answers  and  summarized  discussions  as  they  took  place. 

After  all  the  interview  data  were  collected  (i.e., 
participant  written  responses  and  project  staff  notes) ,  the 
responses  were  entered  into  computer  files,  edited  for 
readability,  and  then  compiled  so  that  similar  answers  did  not 
appear  twice.  The  results  of  the  analysis  are  presented  in 
Appendix  D. 

In  Appendix  D,  the  questions  are  stated,  a  summary  of  the 
responses  is  given,  the  detailed  responses  are  provided,  and  the 
project  staff's  reaction  to  each  set  of  comments  is  stated.  Many 
of  the  testbed  lessons-learned  were  generated  or  reinforced 
during  the  debriefs.  For  this  reason,  the  reader  is  encouraged 
to  closely  examine  Appendix  D. 

In  general,  as  shown  in  Appendix  D,  the  reaction  of  the 
testbed  participants  was  very  positive.  Summary  remarks  from  the 
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aviators  are  provided  in  Table  16  and  from  the  instructors  in 
Table  17. 

Table  16 

Summary  of  Crew  Member  Exit  Interview  Remarks 


Topic 


Remark 


Course 

Administration 


Course  Structure 


Flight  Simulator 


General 

Observations 


Introduce  crew  coordination  from  the 
unit  commander  down. 

Promote  crew  coordination 
standardization  rather  than 
familiarization. 

Retain  both  classroom  instruction  and 
simulator  or  aircraft  application. 
Understood  crew  coordination  basic 
qualities. 

Case  studies  and  video  segments  were 
very  effective. 

Missions  were  realistic  and  demanding. 
Videotape  review  of  performance  is 
absolutely  necessary. 

Glad  to  see  the  emphasis  on  safety. 
Include  nonrated  crew  members. 

It  is  good  to  train  and  evaluate  the 
crew  instead  of  individual  aviators. 

I  realize  that  I  have  a  lot  to 
contribute  as  a  crew  member. 

Got  me  out  of  some  bad  habits. 

Increased  confidence  in  self  and  crew. 
Course  will  probably  save  my  life  in  the 
future . 


Crew  members  appreciated  being  trained  and  evaluated  as  a 
crew  rather  than  as  individual  aviators.  They  emphasized  the 
need  to  get  unit  commanders  involved  at  the  outset  to  realize 
maximum  benefit  from  the  crew  coordination  program.  Crews 
suggested  that  standardized  crew  coordination  training  may  make 


38 


them  more  effective  than  battle-rostering  familiarization.  Crew 
member  participants  said  that  the  case  studies  and  video  segments 
were  very  effective  methods  to  help  them  understand  how  the  Basic 
Qualities  fit  in  with  the  ATMs  and  crew  coordination  program 
objectives.  They  strongly  agreed  on  the  importance  of  including 
both  classroom  instruction  and  hands-on  simulator  or  aircraft 
practice  sessions  as  part  of  the  course.  Though  a  number  of 
crews  had  recently  participated  in  Desert  Storm,  they 
agreed  that  the  simulator  missions  were  realistic  and  demanding. 
They  also  strongly  agreed  that  videotape  review  of  their 
performance  is  essential  to  crew  coordination  training  and 
evaluation.  Crews  spontaneously  shared  the  positive  effect  the 
crew  coordination  project  had  on  mission  safety  and 
effectiveness.  They  recommended  that  the  course  be  implemented 
without  delay. 

Unit  instructor  pilots  and  trainers  who  administered  the 
crew  coordination  course  of  instruction  and  conducted  the 
evaluations  strongly  agreed  that  the  program  is  effective  and 
necessary  to  implement  the  new  Commander's  Guide  (Department  of 
the  Army,  1992,  May)  and  ATMs.  They  echoed  the  crew  member's 
approval  of  the  written  and  video  segment  case  studies  as  being 
effective  in  augmenting  the  classroom  instruction.  As  a  group, 
they  said  that  the  instructor  course  should  require  them  to 
complete  the  same  number  of  hands-on  missions  as  the  crews  they 
teach.  Instructors  strongly  agreed  on  the  importance  of 
videotaping  crew  performance  to  demonstrate  teaching  points  and 
provide  evaluation  feedback.  They  said  that  the  revised  ATM  task 
for  mission  briefing  (Task  1000,  Conduct  crew  mission  briefing) 
(Department  of  the  Army,  1992,  July)  is  very  effective  in 
teaching  and  evaluating  crew  coordination  and  that  an  after¬ 
action  review  (AAR)  task  should  be  developed  for  all  aircraft 
ATMs. 


Testbed  IP  evaluators  were  given  a  job  aid  in  the  form  of  an 
"evaluators  workbook"  (Grubb  et  al.,  1992)  for  use  during  the 
pre-  and  post-training  evaluation  missions.  The  evaluator's 
workbook  was  endorsed  as  applicable  to  all  mission  phases  and  all 
crew  coordination  training  in  simulators  or  aircraft.  Evaluators 
were  comfortable  using  the  Basic  Qualities  and  confident  in  the 
reliability  and  fairness  of  their  ratings.  They  remarked  that 
the  crew  coordination  grade  slips  and  expanded  grading  system 
were  easy  to  use  and  encouraged  their  unreserved  judgements. 
Evaluator  participants  considered  the  crew's  positive  response 
and  improved  performance  to  be  strong  indicators  that  crew 
coordination  can  be  effectively  taught  by  units  in  the  field. 
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Table  17 


Summary  of  Evaluator  and  Trainer  Exit  Interview  Remarks 


Topic 


Remark 


Course  of 
Instruction 


•  Written  and  video  semaent  case  studies  thoroughly  enhanced  training 
effectiveness. 

•  Course  is  effective  and  necessary  to  iapleaent  the  new  Training  Circulars. 

•  Include  all  Student  Course  training  and  evaluation  Missions  in  the  Trainer 
Course. 


Scenarios 


•  Scenarios  were  well  thought  out,  realistic,  and  challenging. 

•  Develop  an  after-action  review  (AAR)  task  W  all  aircraft  ATMs. 

•  Videotape  review  of  crew  perfornance  is  a  MUSt-have  capability. 

•  Include  night  flying  conditions. 

•  Scenarios  addressed  all  Basic  Qualities. 


Evaluator's 

Workbook 


•  Workbook  applies  to  all  phases  of  a  Mission. 

•  Supports  evaluation  of  initial  and  continuation  training  in  simulators  or 
aircraft. 

•  Videotaping  is  essential  to  deMonstrate  teaching  points  and  provide  evaluation 
feeciiack  to  crews. 


Crew  Coordination 
Basic  Qualities 


•  I  used  theM  outside  the  testbed  and  am  convinced  they  are 
coMprehensive  and  distinctly  different. 

•  The  behavioral  anchors  allowed  Me  to  give  fairer  ratings. 

•  The  rating  factors  are  needed  in  the  classroom  instruction. 

•  Behavioral  anchors  and  7'point  rating  scale  are  appropriate  for  the  entire  crew 
coordination  program. 


Grade  Slips  •  Aircrew  Coordination  Training  Grade  Slips  followed  the  already  familiar 

maneuver/procedure  grade  slip. 

•  The  expanded  grading  system  (S*i*,  S,  S-,  U)  allows  IPs  to  identify  strengths  and 
weakn^ses  without  threatening  crew  ratings  or  unit  readiness. 

•  Expanded  rating  system,  if  fielded,  would  allow  me  to  sleep  at  night. 


General 

Observations 


•  Positive  crew  response  and  increased  performance  demonstrate  that  we  can  train 
crew  coordination  in  units. 

•  Enhances  everybody's  ability  to  perform  safely. 

•  Involve  the  chain  of  coamand  from  the  start. 

•  I  could  do  everything  I  needed  to  do  as  an  evaluator. 

•  I  an  comfortable  evaluating  crews  and  confident  that  my  grades  are  fair. 

•  Training  and  evaluations  covered  all  aspects  of  crew  coordination  and  allowed  me 
to  use  my  own  judgment  as  required. 

•  My  evaluations  are  now  crew  rather  than  individual  aviator  centered. 

•I'm  already  using  the  crew  coordination  concepts  during  missions  in  my  mit.  A 
nultiship  NVG  assault  mission  last  week  was  the  best  mission  I  have  flown  in  my 
aviation  career. 
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Moreover,  they  said  that  they  are  better  evaluators  and  that  they 
are  evaluating  crews,  not  individual  crew  members,  as  a  result  of 
participating  in  the  crew  coordination  program.  There  was 
agreement  that  the  course  had  personally  made  them  safer  and  more 
effective  aviators.  The  group  indicated  that  they  were  planning 
to  implement  the  crew  coordination  training  locally  and  that  the 
USAAVNC  should  provide  the  training  on  an  Army-wide  basis  as  soon 
as  possible. 


Operational  Safety  Implications 

The  following  two  sections  discuss  the  potential  impact  of 
crew  coordination  training  on  aviation  safety.  Included  in  the 
discussion  is  crew  coordination  training  impact  on  "marginal” 
crews  and  how  testbed  performance  is  correlated  to  a  USASC 
study . 

Effects  of  the  Crew  Coordination  Training  on  Marginal  Crews 

ARI  expressed  concern  about  the  effects  of  the  crew 
coordination  training  program  on  "marginal"  aviators.  In  the 
past,  an  assumption  was  made  that  a  small  percentage  of  aviators 
cause  most  of  the  human  factors-related  accidents.  This 
assumption  was  the  topic  of  several  discussions  in  the  USAAVNC 
Working  Group  and  has  been  a  longstanding  concern  of  the  USASC. 
This  report  does  not  intend  to  prove  or  disprove  the  assumption. 
However,  if  the  assumption  is  true,  it  is  of  considerable 
interest  to  investigate  the  effects  of  the  training  on  the  lower 
performing  aviators  and  crews.  If  the  data  show  that  crew 
coordination  training  has  a  positive  influence  on  performance  for 
marginal  aviators  (and  the  crews  in  which  they  operate) ,  it  is 
logical  to  conclude  that  the  training  will  have  a  highly  positive 
impact  on  flight  safety. 

The  first  step  in  addressing  ARI's  concern  was  to  determine 
which  crews  participating  in  the  validation  testbed  were 
marginal.  An  analysis  was  done  in  which  the  crews  were  rank- 
ordered  from  low  to  high  on  three  variables:  (1)  Average  Score 
for  the  13  Basic  Quality  ratings,  (2)  Average  Score  for  the  18 
ATM  Tasks,  and  (3)  the  Overall  Grade  for  the  Flight.  On  each  of 
these  variables,  five  crews  ranked  lowest.  These  crews  were 
numbers  7,  11,  13,  14,  and  15.  Next,  post-training  data  was 
generated  on  the  three  variables. 

Review  of  the  data  showed  that  crew  number  14  actually  got 
worse  after  the  training.  This  did  not  seem  right.  Fort 
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Campbell  was  contacted  to  determine  if  perhaps  the  crew  was 
overly  fatigued  after  flying  late  at  night  or  if  some  other 
reason  accounted  for  their  poor  performance.  A  unit  IP  explained 
to  us  that  the  pilot  in  command  (PC)  of  crew  number  14  was  being 
involuntarily  separated  from  the  Army.  We  concluded,  therefore, 
that  it  would  have  been  better  had  he  not  participated  in  the 
testbed  at  all.  Although  we  did  not  go  back  and  remove  that  crew 
from  the  previous  analyses,  we  did  not  include  them  in  the  micro¬ 
analysis  of  marginal  crews.  Thus,  four  crews,  the  lowest  25%, 
were  examined  for  this  analysis. 

Table  18  presents  the  data  for  selected  variables  presented 
in  previous  sections  for  the  four  marginal  crews.  Variables  were 
not  included  in  Table  18  if  there  were  little  or  no  differences 
between  pre-  and  post-training  evaluations.  In  most  of  those 
cases,  a  "ceiling  effect"  existed;  that  is,  most  of  the  scores 
were  very  high  to  start  with. 

Table  18  shows  that  marginal  crews  markedly  improved  from 
the  pre-  to  post-training  evaluations.  The  following  discussion 
focuses  on  only  the  shaded  rows  in  the  table. 

IP  ratings.  Mean  for  all  13  Basic  Quality  Ratings,  Average 
Score  for  the  18  ATM  Tasks,  and  Overall  Grade  for  the  Flight:  IP 
ratings  show  very  high  improvement  across  all  rating  scales. 
Because  these  data  were  generated  using  highly  reliable  measures, 
this  result  may  be  the  most  dependable  measure  of  improved 
performance  for  the  marginal  crews.  All  three  measures  show  that 
the  marginal  crews  improved  to  an  acceptable  or  satisfactory 
level  by  the  time  they  were  evaluated  during  the  post-training 
mission.  In  fact,  the  average  scores  for  the  marginal  crews  are 
very  similar  to  the  average  scores  for  the  entire  testbed  sample. 

Navigation.  Percent  of  Time  Off  Course,  and  Arrived  at 
Correct  LZ:  The  marginal  crews  improved  markedly  in  these  two 
navigation-related  mission  performance  measures.  During  the 
post-training  evaluation,  the  marginal  crews  were  on  course  47% 
more  of  the  time.  Furthermore,  only  two  crews  arrived  at  the 
correct  LZ  during  the  pre-training  ride,  whereas  all  four  crews 
reached  the  correct  LZ  in  the  post-training  ride.  Tactical 
navigation  is  a  highly  crew-coordinated  activity  and  these  crews 
performed  quite  well  in  this  area  in  the  post-training 
evaluation.  Again,  the  crews'  average  score  improved  to  a  level 
that  is  quite  similar  to  the  average  score  for  the  entire  testbed 
sample.  In  fact,  for  these  two  variables,  the  marginal  crews' 
average  was  above  that  for  all  of  the  crews. 
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Table  18 


Pre-  and  Post-training  Scores  for  Marginal  Crews 


Crew  #7 

Crew  #11 

Crew  #13 

Crew  #15 

Average 

Measure 

Pre 

Post 

Pre 

Post 

Pre 

Post 

Pre 

Post 

Pre 

Post 

IP  RATINGS 

mm  W 

z,i 

3.^ 

3,1? 

5,1 

4.4 

2.4 

4.00 

Jtwdte  asoire 
iitr  1&  AW 

iA 

i.t 

ZM 

1,?  ; 

1,3 

St.Z 

firacte 

f»P 

Hilii 

S-- 

ilHIilBi 

s 

NAVIGATION 

#  of  course 

deviations  >500m 

beyond  standard 

2 

1 

5 

1 

3 

2 

4 

1 

3,50 

1.25 

•  %  9i  tim.  9p 

24x7 

SUT 

41.6 

e 

m 

m 

Cetffect  tz? 

not 

1ffe» 

¥66 

%o% 

lOCB^ 

Deviation  in 

seconds  of  LZ 

arrival  time 

N/A 

-21 

62 

-329 

35 

-17 

153 

-380 

N/A 

N/A 

THREAT 

Threat  Encounter 

Rating 

2 

3 

3 

0 

threat 

inoper¬ 

ative 

2 

2 

0 

N/A 

N/A 

EMERGENCIES 

Detect 

minor 

malfunction 

Yes 

Yes 

No 

No 

Yes 

Yes 

No 

Yes 

50% 

100% 

(Continued) 
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Crew  #7 

Crew  #11 

Crew  #13 

Crew  #15 

Average 

Measure 

Pre 

Post 

Pre 

Post 

Pre 

Post 

Pre 

Post 

Pre 

Post 

INSTRUMENT 

RECOVERY 

%  Corn  Rec  Steps 

80 

100 

100 

100 

80 

100 

0 

100 

65% 

100% 

Rating  of 

Marg- 

Supe- 

Marg- 

Marg- 

Good 

Good 

Marg- 

Marg- 

1.25 

1.75 

Approach 

inal 

rior 

inal 

inal 

inal 

inal 

Margina 

Good 

Planning 

1 

#  Altitude 

Deviations 

>  100'  beyond 

standard 

0 

0 

2 

0 

0 

0 

2 

2 

1.3 

.5 

Properly  timed 

inbound  leg 

Yes 

Yes 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

75% 

100% 

OVERALL  MEASURES 

Missism 

gH. 

^IliH 

S 

S- 

liiilli 

*75 

2*5 

imitr  Rating: 

U/S- 

$/S+ 

Pj^cent  of 

33 

m 

33*' 

67 

8D: 

53 

93 

52% 

87% 

mission 

X  PLOT  &  Ext 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

75% 

100% 

Load  Complete? 

#  of  crashes 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

N/A  =  Not  applicable  or  could  not  calculate  because  of  missing  data. 


Overall  measures.  Mission  Threatening  Error  Rating,  and 
Percent  of  Mission  Segments  Accomplished:  All  marginal  crews 
(100%)  were  brought  to  a  level  of  "S”  or  ”8+"  on  the  mission 
threatening  error  rating  scale.  This  is  better  than  the  whole 
group  because  only  75%  of  those  crews  received  an  "S”  or  "8+'' 
rating.  For  the  pre-training  evaluation,  the  marginal  crews 
accomplished  an  average  of  only  52%  of  the  mission  segments;  the 
group  of  16  crews  averaged  66%.  The  percent  of  post-training 
mission  segments  accomplished  by  the  marginal  crews  (87%)  was 
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nearly  identical  to,  and  represents  a  much  larger  gain  than,  the 
average  percentage  accomplished  by  the  whole  group  (86%) . 

Relationship  of  Crew  Coordination  Training  to  Flight  Safety 

Army  aviation  has  an  unrelenting  interest  in  safely 
performing  its  mission.  Certainly,  military  flight  operations 
have  inherent  risks,  but  the  Army  continues  to  upgrade  its 
systems,  doctrine,  and  training  to  mitigate  risks  to  acceptable 
levels.  As  technology  has  improved,  a  smaller  and  smaller 
percentage  of  aviation  accidents  are  caused  by  materiel 
malfunctions.  As  Army  aircraft  have  become  more  reliable,  the 
human  factor  has  increasingly  been  cited  as  a  causal  factor  in 
aviation  accidents. 

A  central  feature  of  the  Army's  crew  coordination  training 
is  that  it  is  designed  to  reduce  the  number  of  accidents.  It  has 
been  noted  in  numerous  accident  investigations  conducted  by  civil 
and  military  agencies  that  the  ability  to  prevent,  or  the 
information  needed  to  avoid  an  accident  is  often  available  in  the 
cockpit.  In  these  instances,  mismanagement  of  resources  and  a 
lack  of  coordinated  activities  led  to  disaster.  Most  of  the 
time,  when  human  factors  is  a  primary  or  significant  contributor 
to  an  accident,  it  is  not  one  miscalculation  or  one  mistake  that 
leads  to  trouble;  instead,  it  is  a  series  of  errors.  In  the 
training  course,  the  chain  of  human  factor  events  leading  to  an 
accident  is  called  the  error  chain. 

Breaking  an  error  chain  before  it  becomes  critical  is 
paramount  to  safe  flight.  Certain  crew  actions  can  be 
implemented  to  break  the  error  chain.  These  actions,  or  crew 
coordination  management  activities  required  to  break  an  error 
chain,  avoid  accidents,  and  effectively  perform  as  an  aircrew  are 
called  the  Crew  Coordination  Objectives  in  the  training  provided 
to  aviators.  The  Crew  Coordination  Objectives,  as  taught  in  the 
course,  are  completely  defined  by  the  aircrew  coordination  Basic 
Qualities,  the  ATM  Aircrew  Coordination  Elements,  and  the  ATM 
Tasks . 

In  a  1991  study,  the  USASC  (Owens,  Robertson,  Thill,  & 
Zeller,  1991)  identified  a  niamber  of  recurring  task  errors 
committed  by  crew  members  who  were  involved  in  over  400  aviation 
accidents  occurring  between  FY84-89.  All  of  the  task  errors 
mentioned  in  the  USASC  report  are  addressed  in  the  crew 
coordination  training  and  evaluated  using  the  Basic  Quality 
ratings  and  ATM  Task  grades.  Descriptions  of  these  task  errors 
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and  their  frequency  of  occurrence  in  the  accident  cases  are  shown 
in  Table  19. 


Table  19 

Human  Error  Patterns  in  Army  Aviation  Accidents  (from  Owens  et 
al.,  1991) 


Percentage  of 


_ Task  error _ accidents^ 

Improper  monitoring  or  scanning  45% 

Improper  decision  or  selection  of  course  of  36% 

action 

Improper  control  action  or  task  completion  24% 

Inadequate  inspection  or  checking  of  SOPs,  ATMs  12% 

Inadequate  communication  of  information  11% 

Misjudge  clearance,  speed,  weight,  size,  9% 

distance ,  time 

Inadequate  planning  and  task  organization  7% 

Failure  to  recognize  or  identify  critical  6% 

condition 

Failure  to  anticipate  upcoming  events  6% 

Misinterpretation  of  communication,  cue,  or  3% 

condition 

Inadequate  improvising,  troubleshooting,  1% 

problem  solving _ _ 


^Percentages  do  not  add  up  to  100%  because  more  than  one  type  of 
task  error  can  be  involved  in  a  given  accident. 


Many  of  the  task  errors  in  Table  19  were  committed  during 
the  execution  of  ATM  Tasks  similar  to  those  performed  during  the 
testbed  missions.  Four  of  the  18  ATM  Tasks  involved  in  the 
testbed  were  the  exact  same  tasks  noted  in  the  USASC  accident 
analysis.  A  fifth  task,  ATM  Task  1071  (Perform  as  a  Crew  Member) 
is  no  longer  an  ATM  Task  in  the  current  revision  of  TC  1-212 
because  crew  coordination  is  now  embedded  in  all  ATM  Tasks. 
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Table  20  shows  the  five  ATM  Tasks  that  appeared  in  both  the 
USASC  analysis  and  the  validation  testbed  scenarios. 


Table  20 

Frequently  Violated  ATM  Procedures  Evaluated  in  the  Testbed 


ATM  Task  #  and  description 

Average 

percentage^ 

1035/2081 

Terrain  Flight 

31% 

1071 

Perform  as  a  Crew  Member 

10% 

1083 

Inadvertent  IMC/VHIRP 

9% 

1028 

VMC  Approach 

6% 

2016 

External  Load  Operations 

3% 

^Average  percentage  value  based  on  frequency  of  occurrence,  total 
accident  cost,  number  of  fatalities,  and  number  of  injuries. 


To  analyze  improvements  due  to  the  testbed,  the  average 
score  on  all  ATM  Tasks  was  substituted  for  ATM  Task  1071,  Perform 
as  a  Crew  Member,  because  (1)  aviators  were  being  evaluated  as 
crews  and  not  as  individuals  and  (2)  crew  coordination 
considerations  are  embedded  in  all  ATM  Tasks.  With  this 
substitution.  Table  21  shows  the  change  in  performance  between 
the  pre-  and  post-training  evaluations  for  each  of  the  above 
listed  frequently  violated  ATM  tasks. 

Performance  improved  across  all  five  tasks,  with  three 
(2081/79,  2016,  and  the  18  Tasks)  of  the  five  grades  showing 
significant  change  after  the  training.  After  the  training, 
performance  was  Satisfactory  for  all  Tasks.  ATM  Task  performance 
can  be  considered  the  “last  link”  in  the  error  chain. 

Breaking  the  chain  at  the  final  point  is  certainly 
important,  but  aviation  error  chains  usually  begin  long  before  an 
ATM  Task  is  performed.  As  discussed  earlier,  the  USAAVNC  Working 
Group  divided  crew  coordination  behaviors  into  micro-  and  macro¬ 
level  behaviors.  The  micro-level  behaviors  are  trained  and 
evaluated  through  the  ATM  Tasks.  Macro-level  behaviors  are 
trained  and  evaluated  through  the  Crew  Coordination  Objectives  as 
defined  by  the  13  Basic  Qualities.  In  terms  of  flight  safety  and 
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Table  21 


Pre-training  to  Post-training  Change  in  Performance^  for 
Frequently  Violated  ATM  Procedures 


UH-60  ATM  Task 

Pre¬ 

training 

Post¬ 

training 

2081 

2079 

Terrain  Flight*^ 

Terrain  Flight  Navigation*^ 

1.4 

ic-k 

2.0 

Average  Score  for  the  18  Tasks 

1.4 

** 

1.9 

1083 

Inadvertent  IMC/VHIRP 

00 

• 

H 

2.1 

1028 

VMC  Approach 

1.9 

2.0 

2016 

External  Load  Operations 

1.4 

kk 

to 

• 

o 

^ATM 

Task  Performance  graded  on  a  S+, 

S,  S-, 

U  (3 

,  2,  1,0) 

scale. 

*^An  average  score  for  ATM  Tasks  2081  and  2079  was  used. 
Also,  the  numbering  and  organization  of  the  new  ATM  changed 
so  that  these  two  tasks  were  the  same  as  1035/2081  in  the 
USASC  study. 

**E  <  .01. 


the  error  chain,  ATM  Task  performance  improvement  addresses  the 
final  link  in  the  error  chain;  however,  the  Crew  Coordination 
Objectives  and  Basic  Qualities  address  the  earlier  links  in  the 
error  chain. 

To  understand  how  the  crew  coordination  training  correlates 
with  the  USASC  accident  data,  an  analysis  was  done  to  correlate 
the  11  USASC  task  errors  (Table  19)  with  the  5  Crew  Coordination 
Objectives.  The  first  step  in  this  analysis  involved  inspecting 
the  overlap  between  each  USASC  task  error  and  the  Crew 
Coordination  Objectives.  For  instance,  improper  monitoring  or 
scanning  errors  relate  directly  to  team  relationships,  exchanging 
mission  information,  and  cross-monitoring  performance.  In  this 
manner,  all  the  USASC  task  errors  were  correlated  with  the  Crew 
Coordination  Objectives.  Table  22  shows  the  relationship  between 
the  USASC  task  errors  and  the  Crew  Coordination  Objectives. 

Note  that  every  task  error  is  addressed  by  at  least  one  of  the 
training  objectives  and,  in  most  cases,  is  covered  by  two  or  more 
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training  objectives.  Instruction  that  is  designed  to  deal  with 
recurring  errors  will  reduce  the  incidence  of  those  errors  and 
should  have  a  positive  effect  on  flight  safety. 

The  bottom  row  of  Table  22  shows  the  average  score  for  the 
Basic  Qualities  composing  each  of  the  Crew  Coordination 
Objectives  for  the  pre-  and  post-training  evaluations.  As  can  be 
seen  in  this  bottom  row,  the  improved  performance  for  each  of  the 
Crew  Coordination  Objectives  is  significant.  For  each  objective, 
performance  improves  from  "marginal”  to  "acceptable"  or  better. 
Presumably  the  margin  of  safety  will  be  improved  likewise. 

The  same  types  of  error  chains  and  task  errors  committed  by 
crews  involved  in  the  USASC  study  were  also  committed  by  the 
testbed  aircrews.  Because  the  mission  was  simulator  based,  no 
injuries  occurred.  Several  errors  observed  during  the  testbed 
were  considered  critical  because  of  their  potential  impact  on 
flight  safety.  Table  23  shows  these  errors  and  the  differences 
in  their  occurrence  from  the  pre-  and  post-training  evaluations. 

There  were  significant  reductions  in  the  occurrence  of  the 
five  critical  errors  shown  in  Table  23  between  the  pre-  and  post¬ 
training  evaluations.  In  two  cases,  the  errors  were  completely 
eliminated;  in  the  other  three,  substantial  reductions  in 
occurrence  were  achieved. 

Although  this  improvement  in  performance  may  not  translate 
directly  to  an  equal  percentage  reduction  in  accidents  and  loss 
of  life  and  property,  there  is  adequate  evidence  to  support  the 
premise  that  flight  safety  is  enhanced  as  a  result  of  the 
training  provided  during  the  testbed.  For  instance,  the 
magnitude  of  the  improvements  in  safety-related  actions  is 
similar  to  those  seen  in  the  mission  performance  enhancements. 
Because  accidents  are  usually  caused  by  a  series  of  errors  (error 
chain)  and  not  just  a  single  error,  these  improvements  should 
result  in  similar  reductions  in  errors  committed  during  flight 
and  a  corresponding  decrease  in  the  overall  nuiaber  of  accidents. 
The  evidence  suggests  that  crew  coordination  training 
significantly  reduces  the  likelihood  of  occurrence  of  the  type  of 
errors  that  the  USASC  has  shown  to  be  major  contributing  factors 
to  previous  accidents. 
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Table  22 


Relationships  Between  the  USASC  Task  Errors  and  USAAVNC  Crew 
Coordination  Objectives 


Nw  Crew  Coordination 

Objectives 

USASC  Task  Errors 

Establish  and 

maintain  team 

relation¬ 

ships 

Mission 

planning  and 

rehearsal 

Establish  and 

maintain 

workload 

levels 

Exchange 

mission 

information 

Cross-monitor 

performance 

Iinproper  nonitoring  or 
scanning 

/ 

/ 

/ 

Improper  decision  or 

selection  of  course  of 

action 

/ 

/ 

/ 

/ 

Improper  control 

action  or  task 

completion 

/ 

/ 

/ 

Inadequate  inspection 

or  checking  of  SOPs, 

ATMs 

/ 

/ 

/ 

Inadequate 

cofimunication  of 

information 

/ 

/ 

/ 

Misjudge  clearance, 
speed,  weight,  size, 
distance,  time 

/ 

/ 

Inadequate  planning 
and  task  organization 

/ 

/ 

/ 

/ 

Failure  to  recognize 

or  identify  critical 

condition 

/ 

/ 

Failure  to  anticipate 
i4)coming  events 

/ 

/ 

/ 

Misinterpretation  of 
comnmication,  cue,  or 

condition 

/ 

(Continued) 
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Nv  Creii  Coordination 

N.  Objectives 

USASC  Task  Erors  \ 

Establish 

and  Maintain 

tean 

relation¬ 

ships 

Mission 

planning  and 

rehearsal 

Establish 

and  Maintain 

workload 

levels 

Exchange 

Mission 

inforMation 

Cross- 

Monitor 

perfonaance 

Inadequate 
iaprovising, 
t  roii>l  eshoot  i  ng, 
problen  solving 

/ 

/ 

/ 

/ 

Pre-  and  Post-Training 

Scores  for  each  Crew 

Pre  Post 

Pre  -♦  Post 

Pre  -♦  Post 

Pre  -►  Post 

Pre  Post 

Coordination 

Objective® 

3.4  4.6 

3.1  -  4.6 

3.2  -  4-3 

3.1  -  4.2 

3-2  4.1 

^Scores  were  determined  by  calculating  the  average  grade  for  the  Basic  Qualities  composing  each 
Crew  Coordination  Objective.  The  Basic  Qualities  were  rated  on  a  seven-point  scale  with  1  =  Very 
Poor,  2  =  Poor,  3  =  Marginal,  4  =  Acceptable,  5  =  Good,  6  =  Very  Good,  and  7  =  Superior. 


Recommendations  for  Fielding  the  Crew  Coordination  Training  and 

Evaluation  Packages 

The  crew  coordination  training  and  evaluation  system  tested 
at  Fort  Campbell  was  very  effective.  The  data  collected  show 
that  the  crew  coordination  training  and  evaluation  system 
improves  attitudes,  positively  affects  behavior,  enhances  mission 
performance,  and  increases  the  margin  of  safety.  In  addition, 
the  instructors  and  aviators  who  participated  in  the  testbed 
attested  to  the  advantages  and  worthiness  of  the  program. 

The  results  reported  in  this  paper  have  been  presented  to 
the  USAAVNC  and  coordinated  at  higher  levels  in  the  Army.  Thus, 
there  are  a  number  of  actions  that  are  being  undertaken. 

1.  The  USAAVNC  is  beginning  to  incorporate  the  training 
into  the  schoolhouse  training  for  the  lERW  and  various  aviator 
qualification  courses  (AQC) . 
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Table  23 


Measured  Improvement  in  Safety  Performance 


Testbed  mission  error 

Pre¬ 

training 

Post¬ 

training 

Reduction 
in  error 

Niamber  of  crews  failing  to 
detect  system  malfunctions 

6 

0 

100% 

Number  of  aircraft  crashes 
due  to  crews  failing  to 
maintain  terrain  and 
obstacle  clearance  during 
terrain  flight 

7 

4 

43% 

Niamber  of  crews  not 
successfully  transitioning 
from  VMC  to  IMG  flight 

2 

0 

100% 

Number  of  crews  violating 
altitude  standards  (>+100') 
during  instrument  recovery 

7 

2 

71% 

Number  of  crew  violating 
minimum  altitude 
restrictions  during 
nonprecision  approach 

6 

1 

83% 

2.  A  USAAVNC  team  of  IPs  is  being  assembled  within  the 
Aviation  Training  Brigade  to  install  this  training  at  each  Army 
simulator  facility  in  the  CONUS,  Europe,  and  Korea  over  the  next 
two  years.  The  team  will  train  the  facility  IPs. 

3.  Video  recording  equipment  is  being  purchased  and 
installed  in  each  of  the  Army's  flight  simulators. 

4.  Reserve  and  National  Guard  forces  will  receive  identical 
training  through  their  simulator  facilities  at  the  Eastern  Army 
Aviation  Training  School  and  the  Western  Army  Aviation  Training 
School . 

5.  The  training  will  be  passed  down  to  the  Army's 
operational  units  as  each  unit  rotates  through  one  of  the 
simulator  facilities  for  their  normal  training  cycle. 
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6.  The  ARIRWARU,  working  with  the  USAAVNC/DES,  will  conduct 
a  follow-on  evaluation  of  aviator  proficiency  to  determine  the 
frequency  of  required  recurrent  training. 

7.  The  Chief  of  Staff  of  the  Army  has  asked  the  Training 
and  Doctrine  Command  (with  technical  assistance  from  the 
ARIRWARU)  to  explore  the  applicability  of  the  USAAVNC  crew 
coordination  training  to  other  crew-served  weapon  systems  in  the 
Army. 


The  testbed  provided  a  number  of  lessons  learned  for  the 
project  staff  and  the  USAAVNC  Working  Group.  Many  of  the  lessons 
learned  at  Fort  Campbell  have  been  incorporated  into  revisions  of 
the  training  and  evaluation  materials  recently  submitted  to  the 
ARI  (Grubb  et  al.,  1992;  Pawlik  et  al.,  1992).  The  process  used 
to  develop  the  materials  is  discussed  in  companion  volumes  (Grubb 
et  al.,  in  preparation;  Pawlik  et  al.,  in  preparation).  Those 
volumes  also  describe  the  products  and  make  recommendations 
specific  to  the  evaluation  or  training  materials.  Most  of  those 
specific  recommendations  are  not  repeated  here.  Only 
recommendations  that  have  broader  implications  for  the  crew 
coordination  project  are  included  in  this  section.  Following  is 
a  list  of  project  recommendations  and  a  short  rationale  for  each. 

1.  The  primary  recommendation  stemming  from  this  project  is 
to  deploy  the  crew  coordination  training  and  evaluation  system. 
The  program  has  significant  military  worth  and  highly  desirable 
aviation  safety  ramifications.  The  new  ATMs,  by  themselves,  do 
not  provide  sufficient  detail  to  implement  the  crew  coordination 
training  and  evaluation  program. 

2.  The  exportable  training  and  evaluation  packages  (Pawlik 
et  al.,  1992;  Grubb  et  al.,  1992)  should  only  be  implemented  in 
units  having  IPs  certified  as  having  completed  Army  Aircrew 
Coordination  Instructor  Training.  To  be  certified,  instructors 
should  be  qualified  IPs  who  have  (a)  completed  the  51-hour 
Instructor  Course,  and  (b)  taught  and  evaluated  the  student 
course  under  the  supervision  of  a  USAAVNC  Trainer. 

3.  The  USASC  and  USAAVNC  should  set  up  a  system  whereby  ATM 
Tasks  that  are  causing  safety  or  performance  problems  are 
identified  and  transmitted  to  the  field.  Unit  Instructor  Pilots 
should  be  given  this  information.  It  was  observed  that  the 
testbed  data  showed  when  IPs  focus  on  selected  ATM  Tasks  crew 
performance  on  those  tasks  improves  significantly. 
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4.  The  Army  should  not  view  crew  coordination  training  as  a 
one-time  train-up  effort.  Reinforcement  of  training  is 
important — practice  and  currency  are  keys  to  its  effective  use. 
The  USAAVNC  should  provide  guidance  to  the  field  on  methods  and 
techniques  to  develop  crew  coordination  continuation  and 
refresher  training. 

5.  Training  of  ''high-risk/low-performer”  aviators  or  crews 
should  be  implemented  as  soon  as  possible.  The  improvement  in 
mission  performance  and  flight  safety  for  low-performing  crews 
will  be  dramatic.  High  risk  aviators  can  be  easily  identified 
through  unit  IPs.  The  ease  with  which  IPs  can  identify 
problematic  aviators  or  crews  was  shown  during  the  1990  testbed. 
The  dramatically  improved  performance  for  low-performing  crews 
was  shown  in  the  1992  testbed. 

6.  An  executive  summary  of  the  program  and  its  benefits 
should  be  written  for  senior  Army  management.  The  summary  could 
take  the  form  of  a  short  written  synopsis,  briefing  charts, 
posters,  a  videotape,  or  a  combination  of  the  four  media.  It 
should  "sell"  the  program.  The  program  requires  a  considerable 
resource  and  cultural  commitment  from  all  levels  of  aviation 
commanders.  An  executive  summary  of  the  program  and  its  benefits 
would  encourage  support  from  the  people  who  will  make  resource 
decisions  and  provide  them  with  information  they  need  to  make  the 
commitment . 

7.  Concentrate  on  training  crew  coordination  behaviors 
through  the  use  of  the  Basic  Qualities  (macro-level  behaviors) 
and  the  ATM  Tasks  (micro-level  behaviors) .  Crew  members  can 
learn  and  IPs  can  evaluate  both  types  of  behaviors.  Furthermore, 
the  testbed  participants  told  us  that  they  appreciated  getting 
all  the  information;  they  preferred  a  more  intellectually 
stimulating  course  instead  of  being  "spoon  fed"  the  material  on 
only  a  basic  level. 

8.  Increase  the  use  of  flight  simulators  to  practice 
tactical  missions  based  on  the  unit  Mission  Essential  Task  List 
(METL) .  Aviators  from  both  the  1990  and  1992  testbeds  commented 
that  they  found  tactical  missions  in  the  simulator  to  be 
challenging  and  instructive.  To  implement  this  recommendation, 
unit  instructors  will  need  to  have  in-depth  training  on  scenario 
development . 
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9.  The  USAAVNC  should  conduct  a  follow-on  evaluation  of 
crew  proficiency  to  determine  the  frequency  of  required  recurrent 
training. 


Recommendations  for  Future  Research 

The  crew  coordination  training  and  evaluation  program 
discussed  in  this  report,  although  successful,  needs  to  be 
considered  in  other  contexts.  A  number  of  areas,  discussed 
below,  would  benefit  from  future  research  and  development  of  the 
program. 

1.  The  crew  coordination  material  needs  to  be  tested  in  the 
AH-64,  CH-47,  UH-1,  and  OH-58  fleets.  The  material  delivered  to 
the  Army  has  been  tested  only  with  the  UH-60.  Each  Army  aircraft 
is  built  and  flown  differently  and  duties  and  responsibilities  in 
the  cockpit  are  different.  In  addition,  several  of  the  Army's 
aircraft  have  nonrated  crew  aboard.  Although  the  project  staff 
believes  that  the  material  delivered  to  the  Army  is  broadly 
applicable  to  other  aircraft,  this  belief  has  not  been 
substantiated . 

2 .  The  crew  coordination  material  needs  to  be  revised  and 
tested  for  aircraft  without  simulators  or  for  units  that  do  not 
have  easy  access  to  high  fidelity  simulators.  Videotaping  is  an 
integral  part  of  the  crew  coordination  training  and  evaluation 
system  that  was  recently  delivered  to  the  Army.  Although 
videotaping  in  a  flight  simulator  presented  minor  technical 
problems  that  were  easily  solved  in  the  current  program, 
videotaping  aboard  an  actual  aircraft  is  a  technical  problem  that 
this  project  did  not  address.  Obtaining  airworthiness 
certification  for  on-board  videotape  equipment  could  be  a  major 
endeavor . 


3 .  The  interactive  effects  of  battle-rostering  and  crew 
coordination  training  need  to  be  assessed.  Both  of  these 
concepts  are  being  deployed  in  the  1992  and  1993  revisions  to  all 
Army  aircraft  training  circulars,  but  battle-rostering  and  crew 
coordination  are  not  synonymous.  Moreover,  the  crew  coordination 
training  has  not  been  designed  to  specifically  address  the  pros 
and  cons  of  battle-rostering.  For  instance,  we  do  not  know 
whether  complacency  becomes  a  factor  in  battle-rostering,  or  how 
often  battle-rostered  crews  should  fly  together. 
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4.  An  automated  method  is  needed  to  collect  statistics  on 
the  crew  coordination  performance  level  for  each  participating 
crew  member  and  unit.  Software  needs  to  be  developed  to 
accommodate  this  need.  If  the  USAAVNC  does  not  develop  the 
software,  each  aviation  unit  will  have  to  do  so  or  forgo  the 
benefits  of  tracking  unit  statistics.  Units  need  to  know  how 
well  aviators,  crews,  and  units  are  performing  mission  essential 
tasks.  Units  also  need  to  know  what  tasks  are  causing  problems 
and  how  well  they  are  performing  ATM  Tasks  in  comparison  to  other 
units.  An  automated  tracking  system  that  will  operate  on  widely 
available  microcomputers  should  be  developed. 

5.  The  training  and  evaluation  packages  (Pawlik  et  al., 
1992;  &  Grubb  et  al.,  1992)  need  to  become  an  integrated  package. 
The  reason  that  separate  packages  were  developed,  one  for 
evaluation  and  one  for  training,  was  largely  due  to  the 
contractual  process.  First,  DRC  was  on  contract  to  develop  the 
field  exportable  evaluation  system;  later,  we  were  on  contract 
for  the  crew  coordination  training  system.  The  result  was  that 
each  area  had  separate  sets  of  deliverable  products.  The 
products  were,  however,  developed  by  the  same  team  and  use  the 
same  notions  of  crew  coordination.  Thus,  it  would  be  feasible  to 
make  the  two  packages  into  one  integrated,  less  complex  package. 

6.  Crew  coordination  skills  would  be  useful  for  cross¬ 
platform  coordination  activities.  Army  aircraft  often  fly 
missions  in  conjunction  with  one  or  more  other  aircraft.  These 
other  aircraft  may  include  similar  types  of  aircraft  or  they  may 
be  aircraft  that  perform  symbiotic  functions  wherein  extensive 
cross-platform  coordination  must  be  implemented;  e.g.,  the  OH-58 
and  the  AH-64.  The  crew  coordination  training  and  evaluation 
system  as  currently  designed  is  for  intra-aircraft  activities. 

It  would  be  useful  to  take  the  same  team-related  concepts  used  in 
crew  coordination  and  extend  them  to  a  larger  team. 

7.  Much  of  the  material  used  in  the  aircrew  coordination 
training  and  evaluation  needs  to  be  transitioned  to  other  crew- 
served  weapon  systems,  such  as  tracked  vehicles,  command  and 
control  centers,  artillery  systems,  etc.  The  program  addressed 
in  the  ARI's  current  research  program  involves  only  aircrew 
coordination;  however,  its  coordination  concepts,  principles,  and 
skills  would  be  the  same  for  other  crew  systems;  only  the 
environment  would  be  different.  Because  the  current  program  has 
proven  to  be  highly  effective,  it  is  likely  that  other  weapon 
systems  would  benefit  as  well. 
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8.  The  crew  coordination  materials  need  to  be  integrated 
into  the  lERW  course.  The  current  program  has  been  developed  for 
use  by  operational  units.  The  problem  of  how  to  best  integrate 
crew  coordination  into  lERW  has  not  been  studied.  Several 
difficulties  are  inherent  to  placing  crew  coordination  training 
into  lERW.  First,  lERW  students  are  primarily  concerned  with 
learning  the  science  of  flying  and  controlling  the  aircraft.  The 
concepts  and  higher  order  cognitive  skills  contained  in  the  crew 
coordination  training  are  unlikely  to  be  well  learned  by  the  lERW 
students  who  are  preoccupied  with  learning  things  of  seemingly 
more  immediate  concern.  Also,  little  time  is  available  in  lERW. 
The  crew  coordination  training  program  requires  18  classroom 
hours  and  several  simulator  missions.  We  have  not  studied  the 
problem  of  what  might  be  eliminated  from  the  crew  coordination 
program  to  fit  into  the  lERW's  very  demanding  and  compressed 
schedule.  After  going  through  a  very  deliberate  development 
process,  project  staff  cannot  determine  what  the  USAAVNC  might 
want  to  drop  because  (a)  all  of  the  material  currently  in  the 
program  has  been  deemed  necessary  by  the  USAAVNC,  and  (b)  the 
validation  testbed  shows  that  the  program  works  well.  A  third 
difficulty  is  that  lERW  students  will  have  problems  practicing 
crew  coordination  skills  because  most  of  their  flight  time  is 
with  an  IP  flying  in  the  other  pilot's  seat  of  the  aircraft. 
Although  the  training  material  urges  IPs  in  this  position  to  role 
play  as  another  pilot,  the  lERW  situation  practically  precludes 
role  playing  for  an  IP;  IPs  need  to  instruct. 

9.  A  study  needs  to  be  undertaken  to  address  the  issue  of 
how  crew  coordination  will  be  introduced  or  integrated  in  the 
USAAVNC  resident  courses.  Examples  of  career  courses  in  which 
crew  coordination  should  be  introduced  include  the  Aviation 
Officer  Basic  and  Advanced,  and  the  Aviation  Senior  Warrant 
Officer  Training  courses.  Examples  of  technical  courses  in  which 
crew  coordination  should  be  integrated  include  the  Initial  Entry 
Rotary  Wing  Aviator,  Aircraft  Qualification,  Instructor  Pilot, 
Rotary  Wing  Refresher,  and  Helicopter  Repairer  courses.  Senior- 
level  courses,  such  as  the  Aviation  Pre-Command  Course,  should 
also  be  given  an  introduction  to  the  aircrew  coordination 
training  program.  If  crew  coordination  training  is  going  to  work 
well  for  the  Army,  then  it  must  be  given  at  all  levels.  Crew 
coordination  training  will  not  work  in  isolation;  it  must  be 
supported  and  encouraged  by  aviation  commanders,  instructors,  and 
crew  members.  The  recommended  study  should  help  the  USAAVNC 
determine  what  aspects  of  the  Aircrew  Coordination  Course  receive 
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emphasis  within  each  of  the  resident  courses.  Subsequent  to  this 
determination,  relevant  training  material  should  be  developed  for 
inclusion  in  each  of  the  courses. 

10.  A  study  should  be  pursued  to  determine  the 
effectiveness  of  using  the  expanded  (S+,  S,  S-,  U)  grading  system 
employed  in  the  testbed.  IPs  from  the  Fort  Campbell  testbed 
liked  this  grading  system  for  several  very  good  reasons  such  as, 
"Yeah,  it  would  help  me  sleep  at  night."  From  a  psychometric 
point  of  view,  IPs  are  clearly  capable  of  grading  ATM  Tasks  on  a 
four-point  scale,  are  comfortable  doing  it,  and  more  information 
is  obtained  in  this  manner.  The  USAAVNC,  on  the  other  hand, 
feels  that  adequate  information  is  gathered  via  the  current  S  or 
U  grading  system. 


Summary  and  Conclusion 

The  ARIRWARU  sponsored  a  very  successful  research  program 
during  1992.  They  worked  closely  with  the  USAAVNC  to  deploy  the 
products  of  its  multiyear  research  program  to  Airmy  aviation's 
operational  community.  The  crew  coordination  training  and 
evaluation  program  as  tested  and  delivered  to  the  Army  can 
significantly  improve  mission  effectiveness  and  enhance  the 
margin  of  aviation  safety.  However,  more  work  needs  to  be  done. 
The  current  work  primarily  involves  deploying  the  training  and 
tailoring  it  to  fit  the  unique  requirements  of  the  Army's  various 
aviation  fleets.  ARIRWARU  has  developed  and  delivered  a  program 
that  works  well.  However,  their  role  in  fielding  the  crew 
coordination  training  and  evaluation  system  will  diminish  because 
much  of  the  work  to  be  done  is  no  longer  considered  to  be 
research  and  development  and  is,  therefore,  outside  of  ARIRWARU' s 
mission. 
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Appendix  A-1 

Army  Aviation  Crewmember  Questionnaire 

Instructions 


The  US  Army  Aviation  Center  (USAAVNC)  and  the  US  Army  Research  Institute  (ARI)  are  researching 
the  area  of  crew  coordination  in  Army  Aviation.  The  goal  of  this  research  is  to  improve  performance 
and  increase  the  margin  of  safety  on  an  Army-wide  basis.  Previous  research  by  other  DoD  services  and 
commercial  aviation  into  the  area  of  crew  coordination  has  contributed  to  substantial  gains  in  both 
performance  and  safety. 

Because  Army  Aviation  is  unique,  much  of  the  information  discovered  by  the  other  services  and  the 
commercial  world  is  not  directly  applicable  to  the  Army  Aviation  environment.  Consequently,  the 
USAAVNC-ARI  research  program  is  designed  to  meet  the  specific  needs  of  Army  aviation.  As  a  result 
of  this  approach,  the  following  actions  are  now  ongoing  or  planned:  Mission  simulations  are  being 
developed  to  stress  aircrew-type  tasks,  enhanced  aircrew  coordination  training  is  being  developed,  the 
US  Army  Safety  Center  is  incorporating  crew  factors  into  the  accident  investigation  process.  Aircrew 
Training  ManuaJs  and  the  annual  proficiency  and  readiness  test  program  are  being  revised,  and  revisions 
to  readiness  reporting  are  being  planned. 

This  Army  Aviation  Crewmember  Questionnaire  has  been  developed  as  part  of  the  USAAVNC-ARI 
research  program  to  obtain  your  opinion  about  crew  operations.  As  an  Army  aviator,  your  participation 
is  essential  to  the  program’s  success.  Your  opinions  are  important  and  will  be  used  to  guide  the  next 
phase  of  the  research  program. 

The  questionnaire  should  take  approximately  20  minutes  to  complete.  The  first  page  of  the  questionnaire 
asks  you  for  background  information  -  please  try  to  be  accurate.  The  next  three  pages  contain  46 
statements  for  which  there  are  no  "right"  or  "wrong"  answers.  We  are  simply  asking  for  your  honest 
opinion  to  each  statement.  Please  consider  each  statement  carefully. 

THANK  YOU  FOR  YOUR  PARTICIPATION. 


IMPORTANT 

The  information  you  provide  in  this  questionnaire  is 
confidential  and  will  be  used  for  research  purposes 
only.  Your  answers  will  neither  be  attributed  to  you 
personally  nor  become  a  part  of  any  personnel  or 
aviation  record  kept  on  you. 
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Army  Aviation  Crewmember  Questionnaire 

I.  Background  Information 

(Please  complete  the  following  information  regarding  your  personal  experiences  and  current  status.) 

1.  Aviation  Experience; 

Lifetime  Flving  Experience  Experience  over  last  6  months 

All  NV  Devices  All  NV  Devices 

Conditions  (e.g.,  NVG)  Conditions  (e.g.,  NVG) 

a.  Primary  acft  hrs.  _  _  _  _ 

b.  RAV  hrs.  _  _  _  _ 

c.  Fixed  Wing  hrs.  _  _  _  _ 

2.  Primary  Aircraft _ (Fill  in  aircraft  designation) 

3.  Current  Rank _ _ 

4.  Current  Unit  (Co/Bn/Rgt) _ 

5.  Time  in  Current  Unit  (months) _ 

6.  Current  Aviator  Readiness  Level  (RL)  12  3  (circle  one  number) 

7.  Current  Crew  Readiness  Level  (CRL)  1  2  (circle  one  number) 

8.  Current  primary  duty  assignment  in  unit  (check  one): 

PC* _  PI _  CP _  CPG _  CE/FE _  AO/AFSO/TO _  OR* _ 

*Note:  PC  includes  IP,  SP,  IE,  UT,  ME,  MP  duty  positions;  OR  includes  gunner  and  flight  medic. 

9.  Are  you  flight  lead  qualified  (circle  one):  Yes  No 

10.  Have  you  had  Aircrew  Coordination  Training?  Y  or  N  (circle  one:  if  yes,  answer  below.) 

Describe  ACT  training  experiences:  Course  title,  location  of  training,  approximate  date,  it  of  hours  of 
instruction,  quality  of  course. 

a.  Experience  ttl:  _ 

b.  Experience  til:  _ 

11.  Cross-indexing  Code  (Note:  Because  the  results  of  this  questionnaire  will  be  correlated  with  other 
measures,  a  social  security  number  is  required.) 

Social  Security  #:  _  Today’s  Date  - 

(day/motyr) 
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II.  Opinion  Survey 

(Please  circle  the  number  on  the  agree-disagree  dimension  that  best  reflects  your  personal  attitude 
toward  each  statement.  There  are  no  "right”  or  "wrong"  answers.  We  are  simply  asking  for  your 
honest  opinions.) 

Strongly  Disagree  Slightly  Neutral  Slightly  Agree  Strongly 
Dis^ree  Disagree  Agree  Agree 


1*  Crewmembers  should  feel  obligated  to  mention  their  own  1  2  3  4  5  6  7 

psychological  stress  or  physical  problems  to  other  crewmembers 
before  or  during  a  mission. 

2.  Crewmembers  should  monitor  each  other  for  signs  of  stress  orl  2  34567 

fatigue  and  should  discuss  the  situation  with  the  affected 

crewmember(s). 

3.  Good  communication  and  crew  coordination  are  as  important  1  2  3  4  5  6  7 

as  technical  proHciency  for  the  safety  of  the  flight. 

4.  Crewmembers  should  be  aware  of  and  sensitive  to  the  personal  1  2  3  4  5  6  7 

problems  of  other  crewmembers* 

5.  The  pilot  flyii^  the  aircraft  should  verbalize  plans  for  1  2  3  4  5  6  7 

procedures  or  maneuvers  and  should  be  sure  that  the 

information  is  understood  and  acknowledged  by  a^ected 
crewmembers. 

6.  Even  when  fatigued,  I  perform  effectively  during  most  critical  1  2  3  4  5  6  7 

flight  maneuvers. 

7.  Pilots-in-command  should  encourage  pilots  and  crew  chiefs  tol  2  34567 

question  procedures  and  flight  proHle  deviations  during  normal 

flight  operations  and  in  emergencies. 

8.  There  are  no  circumstances  where  the  pilot  should  take  the  1  2  34567 

aircraft  controls  without  being  directed  to  do  so  by  the  pilot-in¬ 
command. 

9.  A  debrieHng  and  after  action  review  of  procedures  and  1  2  3  4  5  6  7 

decisions  after  each  mission  are  important  for  developing  and 

maintainiii^  elective  crew  coordination. 

10.  Crew  coordination  is  more  important  under  high  stress  1  2  3  4  5  6  7 

conditions  than  it  is  under  low  stress  conditions. 

11.  Effective  crew  coordination  requires  crewmembers  to  take  into  1  2  3  4  5  6  7 

account  the  personalities  of  other  crewmembers. 
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Strongly  Disagree  Slightly  Neutral  Slightly  Agree  Strongly 
Disagree  Disagree  Agree  Agree 


12.  The  pilot-in-command’s  responsibilities  include  coordinating  1  2  3  4  5  6  7 

inflight  crew  chief  activities* 

13.  Most  crewmembers  are  able  to  leave  personal  problems  behind  1  2  3  4  5  6  7 

when  flying  a  mission. 

14*  My  decision  making  ability  is  as  good  in  emergencies  as  it  is  in  1  2  3  4  5  6  7 

routine  mission  situations* 

15.  The  pilot-in-KTommand  is  solely  responsible  for  leadership  of  the  1  2  3  4  5  6  7 

crew  team. 

16.  PUots  should  consider  crew  chief  questions  and  suggestions.  1  2  3  4  5  6  7 

17.  When  joining  a  unit,  a  new  crewmember  should  not  offer  1  2  3  4  5  6  7 

suggestions  or  opinions  unless  asked* 

18*  Because  crew  chiefs  have  no  pilot  training,  they  should  limit  1  2  3  4  5  6  7 

their  attention  to  their  formally  deHned  crew  chief  duties. 

19.  Pilots-in-command  who  accept  and  implement  suggestions  from  1  2  3  4  5  6  7 

the  crew  lessen  their  stature  and  reduce  their  authority. 

20.  Crewmembers  should  monitor  the  pilot-in-command’s  1  2  3  4  5  6  7 

performance  for  possible  mistakes  and  errors. 

21.  The  best  way  to  correct  an  error  is  to  alert  the  error  maker  sol  2  34567 

that  he  can  correct  the  problem* 

22.  Crewmembers’  errors  and  mistakes  during  the  mission,  1  2  3  4  5  6  7 

including  the  pilot-in-command’s  mistakes,  should  be  a 

signiHcant  part  of  post  flight  crew  discussions. 

23.  The  pilot-in-command  should  seek  advice  from  crewmembers  1  2  3  4  5  6  7 

when  updating  mission  plans. 

24.  Hie  pilot-in-command  should  use  his  crew  to  help  him  maintain  1  2  3  4  5  6  7 

situation  awareness. 

25.  The  pilot-in-command  is  solely  responsible  for  maintaining  1  2  3  4  5  6  7 

awareness  of  crew  capabilities. 

26.  Only  when  the  pilot-in-command  is  overloaded  should  he  pass  1  2  3  4  5  6  7 

workload  to  other  crewmembers. 

27.  Crewmembers  should  be  aware  of  other  crewmembers’  1  2  3  4  5  6  7 

workload. 
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Strongly  Disagree  Slightly  Neutral  Slightly  Agree  Strongly 
Disagree  Disagree  Agree  Agree 


28.  If  a  crewmember  is  having  di^culties  executing  his  1234567 

responsibilities^  other  crewmembers  should  provide  assistance. 

29.  Highly  competent  pilots  do  not  experience  task  overload.  1  2  3  4  5  6  7 

30.  A  crewmember  should  oHer  task  help  to  another  crewmember  1  2  3  4  5  6  7 

only  if  he  is  sure  the  crewmember  needs  it. 

31.  Ibe  pilot-in-command  should  not  get  involved  with  the  1  2  3  4  5  6  7 

execution  of  responsibilities  assigned  to  other  crewmembers. 

32.  Crewmember  task  overload  usually  occurs  because  the  1234567 

crewmember  is  not  very  competent. 

33.  Pilots-in-command  should  employ  the  same  style  of  leadership  1  2  3  4  5  6  7 

in  all  situations  and  with  all  crewmembers. 

34.  Pilot-in-command  instructions  to  other  crewmembers  should  bel  2  34567 

general  and  non-specific  so  that  each  individual  can  practice 

self-management  and  can  develop  individual  skills. 

35.  A  relaxed  attitude  is  essential  for  maintaining  a  cooperative  and  1  2  3  4  5  6  7 

harmonious  cockpit. 

36.  Reprimands  are  more  effective  than  discussions  in  eliminating  al  2  3  4567 

crewmember’s  poor  flying  habit. 

37.  Nonrated  crewmembers  should  be  actively  involved  in  planning  1  2  3  4  5  6  7 

the  mission. 

38.  Understanding  the  commander’s  concept  is  of  minor  1  2  3  4  5  6  7 

importance  to  mission  execution. 

39.  Each  crewmember  should  watch  for  situations  in  which  1  2  3  4  5  6  7 

external  events  limit  others’  performance. 

40.  Thinking  through  difficult  segments,  events,  and  tasks  is  1  234567 

primarily  the  pilot-in-command’s  responsibility. 

41.  My  knowledge  of  unit  SOP  and  aircraft  emergency  procedures  1  2  3  4  5  6  7 

makes  rehearsing  familiar  missions  unnecessary. 

42.  An  essential  element  of  premission  planning  is  discussing  crew  1  2  3  4  5  6  7 

responsibilities  and  required  actions  for  abnormal  events. 

43.  Recent  events  in  my  personal  life  have  little  to  do  with  my  1  2  34567 

performance  as  a  crewmember. 
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Rev.  5 


Strongly 

Disagree 

Disagree 

Slightly 

Disagree 

Neutral 

Slightly 

Agree 

Agree 

Strongly 

Agree 

Crewmembers  should  be  able  to  anticipate  requirements  as  the 
mission  progresses. 

1 

2 

3 

4 

5 

6 

7 

My  individual  performance  is  as  good  in  degraded  systems 
conditions  as  it  is  in  a  '*fiill  up*'  aircraft. 

1 

2 

3 

4 

5 

6 

7 

External  circumstances  require  crewmembers  to  provide 
situational  leadership  for  short  periods  of  time. 

1 

2 

3 

4 

5 

6 

7 
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APPENDIX  A-2 


AKMY  AVIATION  CREWMEMBER  QUESTIONNAIRE  FREQUENCY  TABLES 

Cl  Crewmembers  should  feel  obligated  to  mention  their  own  psychological 


stress  or  physical 

problems 

to  other  crewmembers  before  or 

during  a 

mission. 

Pretraining  Results; 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

5 

15.6 

Agree 

6 

16 

50.0 

Strongly  Agree 

7 

Total 

11 

32 

34.4 

100.0 

Mean  6.188 

Std  dev 

.693 

Valid  cases 

32 

Post- training  Results: 

Response 

Value 

Frequency 

Percent 

Disagree 

2 

1 

3.1 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

18 

56.3 

Strongly  Agree 

7 

Total 

12 

33 

34.4 

100.0 

Mean  6.156 

Std  dev 

.954 

Valid  cases 

32 

C2 

Crewmembers  should 

monitor  each  other  for  signs  of 

stress  or 

fatigue  and 

should  discuss  the 

situation 

with  the  affected  crewmember  ( s )  < 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Neutral 

4 

1 

3.1 

Agree 

6 

17 

53.1 

Strongly  Agree 

7 

14 

43.8 

Total 

32 

100.0 

Mean 

6.375 

Std  dev 

.660 

Valid 

cases 

32 

Post- 

■training  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

1 

3.1 

Agree 

6 

18 

56.3 

Strongly  Agree 

7 

13 

40.6 

Total 

32 

100.0 

Mean 

6.375 

Std  dev 

.554 

Valid 

cases 

32 

C3  Good  comtminicatxon  and  crew  coordination  are  as  iinportant  as  technical 
proficiency  for  the  safety  of  the  flight. 

Pretraining  Results : 


Response 

Value 

Frequency 

Percent 

Slightly  Disagree 

3 

1 

3,1 

Slightly  Agree 

5 

1 

3.1 

Agree 

6 

11 

34.4 

Strongly  Agree 

7 

19 

59.4 

Total 

32 

100.0 

Mean 

6.469 

Std  dev 

.842 

Valid  cases 

32 

Post- 

■'training  Results: 

Response 

Value 

Frequency 

Percent 

Agree 

6 

10 

31.3 

Strongly  Agree 

7 

22 

68.8 

Total 

32 

100.0 

Mean 

6.688 

Std  dev 

.471 

Valid  cases 

32 

C4  CrewzneiKibers  should  be  aware  of  and  sensitive  to  the  personal  problems  of 
other  crewmembers. 


Pre training  Results: 


Response  Value 

Disagree  2 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  5.688  Std  dev 

Post-training  Results: 

Response  Value 

Disagree  2 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  5.938  Std  dev 


Frequency  Percent 

1  3.1 

2  6.3 

7  21.9 

17  53.1 

5  15.6 


32  100.0 

1.03  Valid  cases  32 


Frequency  Percent 

1  3.1 

1  3.1 

4  12.5 

18  56.3 

8  25.0 

32  100.0 

1.01  Valid  cases  32 
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C5  The  pilot  flying  the  aircraft  should  verbalize  plans  for  procedures  or 
maneuvers  and  should  be  sure  that  the  information  is  understood  and 


acknowledged  by 

affected  crewmembers. 

Pre training  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

2 

6,3 

Agree 

6 

15 

46.9 

Strongly  Agree 

7 

15 

46.9 

Total 

32 

100.0 

Mean  6.406 

Std  dev 

.615 

Valid  cases 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Agree 

6 

17 

53.1 

Strongly  Agree 

7 

15 

46.9 

Total 

32 

100.0 

Mean  6,469 

Std  dev 

.507 

Valid  cases 

C6  Even  when  fatigued,  I  perfoirm  effectively  during  most  critical  flight 
maneuvers . 


Pretraining  Results : 


Response  Value 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Total 

Mean  4.625  Std  dev 

Post-training  Results: 

Response  Value 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  3.813  Std  dev 


Frequency  Percent 

1  3.1 

8  25.0 

2  6.3 

12  37.5 

9  28.1 

32  100.0 

1.23  Valid  cases  32 


Frequency  Percent 

9  28.1 

5  15.6 

6  18.8 

8  25.0 

3  9.4 

1  3.1 

32  100.0 

1.49  Valid  cases  32 
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Cl 


Pilots-in-coramand  should  encourage  pilots  and  crew  chiefs  to  question 
procedures  and  flight  profile  deviations  during  normal  flight  operations 
and  in  emergencies . 


Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

1 

3.1 

Disagree 

2 

1 

3.1 

Slightly  Disagree 

3 

4 

12.5 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

13 

40.6 

Strongly  Agree 

7 

10 

31.3 

Total 

32 

100.0 

Mean 

5.531 

Std  dev 

1.66 

Valid  cases 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

1 

3.1 

Disagree 

2 

1 

3.1 

Neutral 

4 

2 

6.3 

Slightly  Agree 

5 

9 

28.1 

Agree 

6 

16 

50.0 

Strongly  Agree 

7 

3 

9.4 

Total 

32 

100.0 

Mean 

5.406 

Std  dev 

1.26 

Valid  cases 

C8  There  are  no  circxunstances  where  the  pilot  should  take  the  aircraft 
controls  without  being  directed  to  do  so  by  the  pilot-in-command. 


Pre training  Results: 

Response 

Strongly  Disagree 
Disagree 

Slightly  Disagree 
Slightly  Agree 
Strongly  Agree 


Mean  1.906 


Value  Frequency 


1  15 

2  11 

3  4 

5  1 

7  1 

Total  32 

Std  dev  1.30 


Percent 

46.9 

34.4 

12.5 
3.1 
3.1 

100,0 

Valid  cases  32 
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Post-training  Results: 

Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Total 

Mean  1.563  Std  dev 


Frequency  Percent 


15 

46.9 

16 

50.0 

1 

3,1 

32 

100.0 

.564  Valid  cases  32 


C9  A  debriefing  and  after  action  review  of  procedures  and  decisions  after 

each  mission  are  important  for  developing  and  maintaining  effective  crew 
coordination . 


Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

18 

56.3 

Strongly  Agree 

7 

12 

37.5 

Total 

32 

100.0 

Mean 

6.313 

Std  dev 

.592 

Valid  cases 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

1 

3.1 

Agree 

6 

13 

40.6 

Strongly  Agree 

7 

18 

56.3 

Total 

32 

100.0 

Mean 

6.531 

Std  dev 

.567 

Valid  cases 

CIO  Crew  coordination  is  more  important  under  high  stress  conditions  than  it 
is  under  low  stress  conditions. 


Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Disagree 

2 

2 

6.3 

Slightly  Disagree 

3 

3 

9.4 

Neutral 

4 

2 

6.3 

Slightly  Agree 

5 

5 

15.6 

Agree 

6 

8 

25.0 

Strongly  Agree 

7 

12 

37.5 

Total 

32 

100.0 

Mean  5.563 

Std  dev 

1.58 

Valid  cases 
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Post-training  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

.  1 

1 

3.1 

Disagree 

2 

4 

12.5 

Slightly  Disagree 

3 

2 

6.3 

Neutral 

4 

3 

9.4 

Slightly  Agree 

5 

6 

18.8 

Agree 

6 

9 

28.1 

Strongly  Agree 

7 

7 

21.9 

Total 

32 

100.0 

Mean  5.000 

Std  dev 

1.79 

Valid 

Cll  Effective  crew  coordination  requires  crewmembers  to 

the  personalities 

of  other  crewmembers. 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Disagree 

2 

1 

3.1 

Slightly  Disagree 

3 

1 

3.1 

Neutral 

4 

3 

9.4 

Slightly  Agree 

5 

6 

18.8 

Agree 

6 

14 

43.8 

Strongly  Agree 

7 

7 

21.9 

Total 

32 

100.0 

Mean  5.625 

Std  dev 

1.21 

Valid  ( 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

5 

15.6 

Agree 

6 

22 

68.8 

Strongly  Agree 

7 

4 

12.5 

Total 

32 

100.0 

Mean  5.906 

Std  dev 

.641 

Valid  ( 

32 


32 


32 
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C12  The  pilot-in*”Conimand.' s  responsibilities  include  coordinating  inflight 
crew  chief  activities. 

Pretraining  Results: 


Response 

Slightly  Disagree 
Neutral 

Slightly  Agree 
Agree 

Strongly  Agree 


Mean  6.000 

Post -training  Results: 

Response 

Slightly  Disagree 
Neutral 
Slightly  Agree 
Agree 

Strongly  Agree 


Mean  5.875 


Value  Frequency 


3  1 

4  1 

5  3 

6  19 

7  8 

Total  32 

Std  dev  .880 


Value  Frequency 


3  1 

4  2 

5  3 

6  20 

7  6 

Total  32 

Std  dev  .907 


Percent 

3.1 

3.1 

9.4 

59.4 

25.0 

100.0 

Valid  cases  32 


Percent 

3.1 

6.3 

9.4 
62.5 
18.8 

100.0 

Valid  cases  32 


Cl 3  Most  crewmezhbers  are  able  to  leave  personal  problems  behind  when  flying 

a  mission. 


Pretraining  Results: 


Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  4.375  Std  dev 


Frequency  Percent 

1  3.1 

4  12.5 

3  9.4 

10  31.3 

3  9.4 

10  31.3 

1  3.1 

32  100.0 

1.56  Valid  cases  32 
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Post-training  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

2 

6.3 

Disagree 

2 

7 

21.9 

Slightly  Disagree 

3 

8 

25.0 

Neutral 

4 

4 

12.5 

Slightly  Agree 

5 

5 

15.6 

Agree 

6 

6 

18.8 

Total 

32 

100.0 

Mean  3.656 

Std  dev 

1.59 

Valid 

Cl 4  My  decision  zoaking  ability  is 

mission  situations. 

as  good  in  emergencie 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Disagree 

2 

1 

3.1 

Slightly  Disagree 

3 

5 

15.6 

Neutral 

4 

6 

18.8 

Slightly  Agree 

5 

3 

9.4 

Agree 

6 

12 

37.5 

strongly  Agree 

7 

5 

15.6 

Total 

32 

100.0 

Mean  5.094 

Std  dev 

1.46 

Valid 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Disagree 

2 

4 

12.5 

Slightly  Disagree 

3 

3 

9.4 

Neutral 

4 

3 

9.4 

Slightly  Agree 

5 

3 

9.4 

Agree 

6 

16 

50.0 

Strongly  Agree 

7 

3 

9.4 

Total 

32 

100.0 

Mean  5.031 

std  dev 

1.59 

Valid 

32 


32 


32 
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CIS  The  pilot-in-coiranand  is  solely  responsible  for  leadership  of  the  crew 
teeun. 


Pre training  Results : 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

3 

9.4 

Disagree 

2 

7 

21,9 

Slightly  Disagree 

3 

7 

21.9 

Slightly  Agree 

5 

4 

12.5 

Agree 

6 

7 

21.9 

Strongly  Agree 

7 

4 

12.5 

Total 

32 

100.0 

Mean 

4.000 

Std  dev 

2.04 

Valid  cases 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

5 

15.6 

Disagree 

2 

7 

21.9 

Slightly  Disagree 

3 

4 

12.5 

Neutral 

4 

2 

6.3 

Slightly  Agree 

5 

5 

15.6 

Agree 

6 

7 

21.9 

Strongly  Agree 

7 

2 

6.3 

Total 

32 

100.0 

Mean 

3.750 

Std  dev 

2.01 

Valid  cases 

C16  Pilots  should  consider  crew  chief  questions  and  suggestions. 


Pretraining  Results: 

Response  Value 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  6.500  Std  dev 

Post-training  Results: 

Response  Value 

Disagree  2 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  6.219  Std  dev 


Frequency  Percent 

1  3.1 

14  43.8 

17  53.1 

32  100.0 

.568  •  Valid  cases  32 


Frequency  Percent 

1  3.1 

1  3.1 

18  56.3 

12  37.5 

32  100.0 

.941  Valid  cases  32 
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C17 

When  joining  a  unit,  a  new  crewmember  should  not  offer  suggestions 

opinions  unless  asked. 

Pretrainxng  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

10 

31.3 

Disagree 

2 

14 

43.8 

Slightly  Disagree 

3 

2 

6.3 

Neutral 

4 

4 

12.5 

Slightly  Agree 

5 

1 

3.1 

Agree 

6 

1 

3.1 

Total 

32 

100.0 

Mean 

2.219 

Std  dev 

1.28 

Valid  cases 

32 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

6 

18.8 

Disagree 

2 

20 

62.5 

Slightly  Disagree 

3 

3 

9.4 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

2 

6.3 

Total 

32 

100.0 

Mean 

2.156 

Std  dev 

.987 

Valid  cases 

32 

C18  Because  crew  chiefs  have  no  pilot  training,  they  should  limit 

their 

attention  to  their 

formally  defined  crew  chief  duties. 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

8 

25.0 

Disagree 

2 

13 

40.6 

Slightly  Disagree 

3 

7 

21.9 

Slightly  Agree 

5 

3 

9.4 

Agree 

6 

Total 

1 

32 

3.1 

100,0 

Mean  2.375 

Std  dev 

1.31 

Valid  cases 

32 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

6 

18.8 

Disagree 

2 

16 

50.0 

Slightly  Disagree 

3 

6 

18.8 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

Total 

3 

32 

9.4 

100.0 

Mean  2.344 

Std  dev 

1.12 

Valid  cases 

32 
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Cl 9  Pilots -in-command  who  accept  and  implement  suggestions  from  the  crew 

lessen  their  stature  and  reduce  their  authority. 


Pre training  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

17 

53.1 

Disagree 

2 

11 

34.4 

Slightly  Disagree 

3 

1 

3.1 

Neutral 

4 

2 

6.3 

Agree 

6 

1 

3.1 

Total 

32 

100.0 

Mean 

1.750 

Std  dev 

1.13 

Valid  cases 

32 

Post- 

’training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

14 

43.8 

Disagree 

2 

18 

56.3 

Total 

32 

100.0 

Mean 

1.563 

Std  dev 

.504 

Valid  cases 

32 

C20  Crewmembers  should 

.  monitor  the  pilot-in- 

-command's  performance 

for 

possible  mistakes 

and  errors. 

Pre training  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Disagree 

3 

2 

6.3 

Neutral 

4 

3 

9.4 

Slightly  Agree 

5 

6 

18.8 

Agree 

6 

14 

43.8 

Strongly  Agree 

7 

Total 

7 

32 

21.9 

100.0 

Mean  5.656 

Std  dev 

1.12 

Valid  cases 

32 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

3 

9.4 

Agree 

6 

22 

68.8 

Strongly  Agree 

7 

Total 

6 

32 

18.8 

100.0 

Mean  6.031 

Std  dev 

.647 

Valid  cases 

32 
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C21  The  best  way  to  correct  an  error  is  to  alert  the  error  maker  so  that  he 
can  correct  the  problem. 


Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

3 

9.4 

Agree 

6 

18 

56.3 

Strongly  Agree 

7 

Total 

10 

32 

31.3 

100.0 

Mean  6.156 

Std  dev 

.723 

Valid  cases 

32 

Post- training  Results : 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

20 

62.5 

Strongly  Agree 

7 

Total 

10 

32 

31.3 

100.0 

Mean  6.250 

Std  dev 

.568 

Valid  cases 

32 

C22  Crewmembers '  errors 

;  3Lnd  mistakes  during 

the  mission,  including  the 

pilot-in-command*  s 
crew  discussions . 

mistakes. 

should  be  a 

L  significant  part 

of  post  flight 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

18 

56.3 

Strongly  Agree 

7 

Total 

12 

32 

37.5 

100.0 

Mean  6.313 

Std  dev 

.592 

Valid  cases 

32 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

4 

12.5 

Agree 

6 

13 

40.6 

Strongly  Agree 

7 

Total 

15 

32 

46.9 

100.0 

Mean  6.344 

Std  dev 

.701 

Valid  cases 

32 
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C23 

The  pilot-in-coromand  should 

seek  advice 

from  crewmembers  when 

updating 

mission  plans. 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Disagree 

3 

1 

3.1 

Neutral 

4 

2 

6.3 

Slightly  Agree 

5 

10 

31.3 

Agree 

6 

9 

28.1 

Strongly  Agree 

7 

10 

31.3 

Total 

32 

100.0 

Mean 

5.781 

Std  dev 

1.07 

Valid  cases 

32 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

5 

15.6 

Agree 

6 

21 

65.6 

Strongly  Agree 

7 

6 

18.8 

Total 

32 

100.0 

Mean 

6.031  ^ 

Std  dev 

.595 

Valid  cases 

32 

C24 

The  pilot-in-command  should  i 

use  his  crevi 

r  to  help  him  maintain 

situation 

awareness . 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

1 

3.1 

Agree 

6 

11 

34.4 

Strongly  Agree 

7 

20 

62.5 

Total 

32 

100.0 

Mean 

6.594 

Std  dev 

.560 

Valid  cases 

32 

Post- 

•training  Results: 

Response 

Value 

Frequency 

Percent 

Agree 

6 

13 

40.6 

Strongly  Agree 

7 

19 

59.4 

Total 

32 

100.0 

Mean 

6.594 

Std  dev 

.499 

Valid  cases 

32 
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C25  The  pilot-in- command  is  solely  responsible  for  maintaining  awareness  of 
crew  capabilities. 

Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

2 

6.3 

Disagree 

2 

15 

46.9 

Slightly  Disagree 

3 

8 

25.0 

Neutral 

4 

2 

6.3 

Slightly  Agree 

5 

3 

9.4 

Agree 

6 

1 

3.1 

Strongly  Agree 

7 

1 

3.1 

Total 

32 

100.0 

Mean  2.875 

Std  dev 

1.40 

Valid 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

6 

18.8 

Disagree 

2 

16 

50.0 

Slightly  Disagree 

3 

5 

15.6 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

3 

9.4 

Strongly  Agree 

7 

1 

3.1 

Total 

32 

100.0 

Mean  2.469 

Std  dev 

1.39 

Valid  1 

C26  Only  when  the  pilot-in-command  is  overloaded  should 

other  crewmembers. 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

15 

46.9 

Disagree 

2 

13 

40.6 

Slightly  Disagree 

3 

1 

3.1 

Agree 

6 

2 

6.3 

Strongly  Agree 

7 

1 

3.1 

Total 

32 

100.0 

Mean  1.969 

Std  dev 

1.53 

Valid  c 

32 


32 


32 
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Post-training  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

13 

40,6 

Disagree 

2 

15 

46.9 

Slightly  Disagree 

3 

1 

3.1 

Neutral 

4 

1 

3.1 

Strongly  Agree 

7 

Total 

2 

32 

6.3 

100.0 

Mean  2.000 

Std  dev 

1.48 

Valid  cases 

C27  CrewmeiDbers  should  be  aware  of  other  crewmembers*  workload. 
Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

1 

3.1 

Slightly  Agree 

5 

4 

12.5 

Agree 

6 

17 

53.1 

Strongly  Agree 

7 

10 

31.3 

Total 

32 

100.0 

Mean 

6.031 

Std  dev 

1.12 

Valid  cases 

Post- 

■training  Results: 

Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

20 

62.5 

Strongly  Agree 

7 

10 

31.3 

Total 

32 

100.0 

Mean 

6.250 

Std  dev 

.568 

Valid  cases 

C28  If  a  crewmember  is  having  difficulties  executing  his  responsibilities, 
other  crewmembers  should  provide  assistance. 


Pretraining  Results: 


Response  Value 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  6.125  Std  dev 


Frequency  Percent 

1  3.1 

3  9.4 

19  59.4 

9  28.1 

32  100.0 

.707  Valid  cases  32 
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Post-training  Results: 


Response 

Value 

Frequency 

Percent 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

18 

56.3 

Strongly  Agree 

7 

12 

37.5 

Total 

32 

100.0 

Mean  6.313 

Std  dev 

.592 

Valid  cases 

C29  Highly  coir5>etent  pilots  do  not  experience  task  overload. 


Pre training  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

18 

56.3 

Disagree 

2 

12 

37.5 

Slightly  Disagree 

3 

1 

3.1 

Neutral 

4 

1 

3.1 

Total 

32 

100.0 

Mean 

1.531 

Std  dev 

.718 

Valid  cases 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

18 

56.3 

Disagree 

2 

11 

34.4 

Slightly  Disagree 

3 

1 

3.1 

Neutral 

4 

2 

6.3 

Total 

32 

100.0 

Mean 

1.594 

Std  dev 

.837 

Valid  cases 

C30  A  crewmember  should  offer  task  help  to  another  crewmember  only  if  he  is 
sure  the  crewmember  needs  it. 


Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

5 

15.6 

Disagree 

2 

15 

46.9 

Slightly  Disagree 

3 

8 

25.0 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

2 

6.3 

Strongly  Agree 

7 

Total 

1 

32 

3.1 

100.0 

Mean  2.500 

Std  dev 

1.29 

Valid  cases 
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Post-training  Results: 


Response 

Strongly  Disagree 
Disagree 

Slightly  Disagree 
Neutral 

Slightly  Agree 
Strongly  Agree 


Mean  2.250 


Value  Frequency 


1  8 

2  15 

3  6 

4  1 

5  1 

7  1 

Total  32 

Std  dev  1.27 


Percent 

25.0 

46.9 

18.8 

3.1 

3.1 

3.1 

100.0 

Valid  cases  32 


C31  The  pilot-in-coxnmand  should  not  get  involved  with  the  execution  of 
responsibilities  assigned  to  other  crewxneirbers 


Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

3 

9.4 

Disagree 

2 

14 

43.8 

Slightly  Disagree 

3 

9 

28.1 

Neutral 

4 

3 

9.4 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

1 

3.1 

Total 

32 

100.0 

Mean  2.688 

Std  dev 

1.17 

Valid  cases 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

6 

18.8 

Disagree 

2 

12 

37.5 

Slightly  Disagree 

3 

9 

28.1 

Slightly  Agree 

5 

5 

15.6 

Total 

32 

100.0 

Mean  2.563 

Std  dev 

1.26 

Valid  cases 

32 


32 


C32  Crewmember  task  overload  usually  occurs 


very  competent. 
Pretraxnxng  Results: 

Response 

Strongly  Disagree 
Disagree 

Slightly  Disagree 
Neutral 
Slightly  Agree 


Mean  2.281 


Value  Frequency 


1  7 

2  17 

3  2 

4  4 

5  2 

Total  32 

Std  dev  1.14 


Post-training  Results: 

Response 

Strongly  Disagree 
Disagree 

Slightly  Disagree 
Slightly  Agree 


Value  Frequency 


1  9 

2  16 

3  6 

5  1 

Total  32 


Mean 


2.000  Std  dev 


.880 


because  the  crewmember  is  not 


Percent 

21.9 

53.1 

6.3 

12.5 

6.3 


100.0 

Valid  cases  32 


Percent 

28.1 

50.0 

18.8 

3.1 

100.0 


Valid  cases  32 


C33  Pilots-in-command  should  employ  the  same  style  of  leadership  in  all 
situations  and  with  all  crewmembers . 

Pretraining  Results: 


Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Agree  6 

Strongly  Agree  7 

Total 

Mean  2.000  Std  dev 


Frequency  Percent 

6  18.8 

12  37.5 

6  18.8 

2  6.3 

5  15.6 

1  3.1 

32  100.0 

1.78  Valid  cases  32 


A~2~18 


Post-training  Results: 


Response 

Strongly  Disagree 
Disagree 

Slightly  Disagree 
Neutral 

Slightly  Agree 
Agree 

Strongly  Agree 


Mean  2.844 


Value  Frequency 


1  5 

2  15 

3  4 

4  1 

5  3 

6  3 

7  1 

Total  32 

Std  dev  1.70 


Percent 

15.6 

46.9 

12.5 

3.1 

9.4 

9.4 

3.1 

100.0 

Valid  cases  32 


C34  Pilot-in-command  instructions  to  other  crewmembers  should  be  general  and 
non-specific  so  that  each  individual  can  practice  self -management  and 
can  develop  individual  skills. 


Pretraining  Results: 


Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Strongly  Agree  7 

Total 

Mean  2.813  Std  dev 

Post- training  Results : 

Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Total 

Mean  2.719  Std  dev 


Frequency  Percent 

6  18.8 

9  28.1 

8  25.0 

5  15.6 

3  9.4 

1  3.1 

32  100.0 

1.44  Valid  cases  32 


Frequency  Percent 

5  15.6 

15  46.9 

3  9.4 

4  12.5 

3  9.4 

2  6.3 

32  100.0 

1.46  Valid  cases  32 
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C35  A  relaxed  attitude  is  essential  for  maintaining  a  cooperative  and 
harmonious  cockpit. 


Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

1 

3.1 

Disagree 

2 

1 

3.1 

Neutral 

4 

6 

18.8 

Slightly  Agree 

5 

6 

18.8 

Agree 

6 

11 

34,4 

Strongly  Agree 

7 

7 

21.9 

Total 

32 

100.0 

Mean  5.375 

Std  dev 

1.45 

Valid  cases 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

1 

3.1 

Slightly  Disagree 

3 

1 

3.1 

Neutral 

4 

6 

18.8 

Slightly  Agree 

5 

4 

12.5 

Agree 

6 

16 

50.0 

Strongly  Agree 

7 

4 

12.5 

Total 

32 

100.0 

Mean  5.375 

Std  dev 

1.31 

Valid  cases 

C36  Reprimands  are  more  effective  than  discussions  in  eliminating  a 
crewmember's  poor  flying  habit. 


Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

9 

28.1 

Disagree 

2 

17 

53.1 

Slightly  Disagree 

3 

2 

6.3 

Neutral 

4 

2 

9.4 

Slightly  Agree 

5 

1 

3.1 

Total 

32 

100.0 

Mean  2.063 

Std  dev 

1.01 

Valid  cases 

Post-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

14 

43.8 

Disagree 

2 

12 

37.5 

Slightly  Disagree 

3 

4 

12.5 

Neutral 

4 

2 

6.3 

Total 

32 

100.0 

Mean  1.813 

Std  dev 

.896 

Valid  cases 
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C37  Nonrated  crewxneinbers  should  be  actively 
mission. 


Pre training  Results: 


Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  4.250  Std  dev 

Post-training  Results: 

Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  5.031  Std  dev 


Frequency 

3 

3 

4 

5 

10 
4  ■ 

3 

32 

1.74 


Frequency 

1 

5 

1 

2 

4 
14 

5 

32 

1.80 


involved  in  planning  the 


Percent 


9.4 

9.4 

12.5 

15.6 
31.3 
12.5 

9.4 


100.0 


Valid  cases  32 


Percent 

3.1 

15.6 

3.1 

6.3 

12.5 
43.8 

15.6 

100.0 


Valid  cases  32 


C38  Understanding  the  commander's  concept  is  of  minor  importance  to  mission 
execution . 


Pretraining  Results: 

Response 

Strongly  Disagree 
Disagree 

Slightly  Disagree 
Neutral 


Value  Frequency  Percent 

1  11  34.4 

2  17  53.1 

3  3  9.4 

4  1  3.1 

Total  32  100.0 


Mean 


1.813  Std  dev 


.738  Valid  cases  32 
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Post-training  Results : 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

8 

25.0 

Disagree 

2 

17 

53.1 

Slightly  Disagree 

3 

5 

15.6 

Slightly  Agree 

5 

1 

3.1 

Strongly  Agree 

7 

Total 

1 

32 

3.1 

100.0 

Mean  2.156 

Std  dev 

1.22 

Valid  cases 

C39  Each  crewmember  should  watch  for  situations  in  which  external  events 
limit  others '  performance . 

Pretraining  Results : 


Response  Value 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  5.500  Std  -dev 

Post-training  Results: 

Response  Value 

Disagree  2 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  5.938  Std  dev 


Frequency  Percent 

1  3.1 

1  3.1 

2  6.3 

8  25.0 

17  53.1 

3  9.4 

32  100.0 

1.07  Valid  cases  32 


Frequency  Percent 

1  3.1 

2  6.3 

25  78.1 

4  12.5 

32  100.0 

.840  Valid  cases  32 
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C40 

Thinking  through  difficult  segments,  events,  and  tasks  is 

primarily  the 

pilot-in-command*  s 

responsibility . 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

8 

25.0 

Disagree 

2 

10 

31.3 

Slightly  Disagree 

3 

6 

18.8 

Slightly  Agree 

5 

6 

18.8 

Agree 

6 

2 

6.3 

Total 

32 

100.0 

Mean 

2.750 

Std  dev 

1.62 

Valid 

cases 

32 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

5 

15.6 

Disagree 

2 

14 

43.8 

Slightly  Disagree 

3 

7 

21.9 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

3 

9.4 

Total 

32 

100.0 

Mean 

2.688 

Std  dev 

1.46 

Valid 

cases 

32 

C41 

My  knowledge  of  unit  SOP  and 

aircraft  emergency  procedures 

makes 

rehearsing  familiar  missions 

unnecessary. 

Pretraining  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

12 

37.5 

Disagree 

2 

16 

50.0 

Slightly  Disagree 

3 

3 

9.4 

Slightly  Agree 

5 

1 

3.1 

Total 

32 

100.0 

Mean 

1.813 

Std  dev 

0.85 

Valid  cases 

32 

Post- 

■training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

10 

31.3 

Disagree 

2 

18 

56.3 

Slightly  Disagree 

3 

3 

9.4 

Slightly  Agree 

5 

1 

3.1 

Total 

32 

100.0 

Mean 

1.875 

Std  dev 

.833 

Valid 

cases 

32 
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C42  An  essential  element  of  premission  planning  is  discussing  crew 
responsibilities  and  required  actions  for  abnormal  events 


Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Disagree 

2 

1 

3.1 

Slightly  Disagree 

3 

1 

3.1 

Slightly  Agree 

5 

2 

6.3 

Agree 

6 

16 

50.0 

Strongly  Agree 

7 

12 

37.5 

Total 

32 

100.0 

Mean 

6.094 

Std  dev 

1.11 

Valid  cases 

32 

Post- 

-training  Results: 

Response 

Value 

Frequency 

Percent 

Agree 

6 

14 

43.8 

Strongly  Agree 

7 

18 

56.3 

Total 

32 

100.0 

Mean 

6.563 

Std  dev 

.504 

Valid  cases 

32 

C43  Recent  events  in  my  personal  life  have  little  to  do  with  my  performance 
as  a  crewmember. 


Pre training  Results: 


Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  3.563  Std  dev 

Post-training  Results: 

Response  Value 

Strongly  Disagree  1 

Disagree  2 

Slightly  Disagree  3 

Neutral  4 

Slightly  Agree  5 

Agree  6 

Strongly  Agree  7 

Total 

Mean  3.625  Std  dev 


Frequency  Percent 

5  15.6 

7  21.9 

6  18.8 

4  12.5 

2  6.3 

5  15.6 

3  9.4 

32  100.0 

1.98  Valid  cases  32 


Frequency  Percent 

6  18.8 

5  15.6 

6  18.8 

3  9.4 

5  15.6 

4  12.5 

3  9.4 

32  100.0 

1.99  Valid  cases  32 
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C44 


Crewmembers  should  be  able  to  anticipate  requirements  as  the  mission 
progresses . 


Pretraining  Results: 


Response 

Value 

Frequency 

Percent 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

7 

21.9 

Agree 

6 

20 

62.5 

Strongly  Agree 

7 

4 

12.5 

Total 

32 

100.0 

Mean 

5.844 

Std  dev 

.677 

Valid  cases 

Post- 

'training  Results: 

Response 

Value 

Frequency 

Percent 

Neutral 

4 

1 

3.1 

Slightly  Agree 

5 

6 

18.8 

Agree 

6 

18 

56.3 

Strongly  Agree 

7 

7 

21.9 

Total 

32 

100.0 

Mean 

5.969 

Std  dev 

.740 

Valid  cases 

C45  My  individual  performance  is  as  good  in  degraded  systems  conditions  as 
it  is  in  a  "full  up”  aircraft. 

Pre training  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

1 

3.1 

Disagree 

2 

5 

15.6 

Slightly  Disagree 

3 

7 

21,9 

Neutral 

4 

4 

12.5 

Slightly  Agree 

5 

12 

37.5 

Agree 

6 

2 

6.3 

Strongly  Agree 

7 

Total 

1 

32 

3.1 

100.0 

Mean  3.969 

Std  dev 

1.44 

Valid  cases 
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Post- training  Results: 


Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

2 

6.3 

Disagree 

2 

8 

25.0 

Slightly  Disagree 

3 

4 

12.5 

Neutral 

4 

2 

6.3 

Slightly  Agree 

5 

9 

28.1 

Agree 

6 

6 

18.8 

Strongly  Agree 

7 

1 

3.1 

Total 

32 

100.0 

Mean  3.938 

Std  dev 

1.44 

Valid  cases  32 

C46  External  circumstances  require  crewmembers  to  provide  situational 

leadership  for  short  periods 

of  time . 

Pre training  Results: 

Response 

Value 

Frequency 

Percent 

Strongly  Disagree 

1 

1 

3.1 

Neutral 

4 

7 

21.9 

Slightly  Agree 

5 

8 

25.0 

Agree 

6 

15 

46.9 

Strongly  Agree 

7 

Total 

1 

32 

3.1 

100.0 

Mean  5.188 

Std  dev 

1.14 

Valid  cases 

32 

Post-training  Results : 

Response 

Value 

Frequency 

Percent 

Neutral 

4 

2 

6.3 

Slightly  Agree 

5 

4 

12.5 

Agree 

6 

21 

65.6 

Strongly  Agree 

7 

Total 

5 

32 

15,6 

100.0 

Mean  5.906 

Std  dev 

.734 

Valid  cases 

32 
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Appendix  B 

Aircrew  Coordination  Evaluation  (ACE)  Checklist  and  Basic 
Qualities^  and  Frequency  Tables 


Note:  The  Aircrew  Coordination  Evaluation  (ACE)  Checklist, 

developed  for  use  in  the  Field  Exportable  Evaluation  Package,  is 
presented  here  as  an  index  to  the  crew  coordination  Basic 
Qualities 


B-1 


Appendix  B-1 

Behavioral  Anchored  Ratings 


AIRCREW  COORDINATION  EVALUATION  (ACE)  CHECKLIST 

For  use  of  this  form,  see  Aircrew  Coordination  Exportable  Evaluation 
Package  for  Army  Aviation. 


CREW  COORDINATION  BASIC  QUAUTIES 


1  Establish  and  maintain  flight  team  leadership  and  crew  climate 
(Crew  Climate) 


2  Premission  planning  and  rehearsal  accomplished  (Plan  Rehearse) 


3  Application  of  appropriate  decision  making  techniques  (Decision  Tech) 


Prioritize  actions  and  distribute  workload  (Workload) 


5  Management  of  unexpected  events  (Unexp  Events) 


6  Statements  and  directives  clear,  timely,  relevant,  complete,  and  verified 
(Info  Xfer) 


Maintenance  of  mission  situational  awareness  (Sit  Aware) 


8  Decisions  and  actions  communicated  and  acknowledged  (Comm/Ack) 


9  Supporting  information  and  actions  sought  from  crew  (Info  Sought) 


10  Crewmember  actions  mutually  cross-monitored  (Cross  Monitor) 


1 1  Supporting  information  and  actions  offered  by  crew  (Info  Offered) 


12  Advocacy  and  assertion  practiced  (Advoc/Assert) 


13  Crew-level  after-action  reviews  accomplished  (AAR) 


Evaluator's  Signature: 


Notes: 

Consult  the  behavioral  anchored  rating  guidance.  Enter  a  summary  rating  (1,  2  ...  7)  in  the  rating 
block  for  each  Basic  Quality.  Refer  to  the  rating  scale  below. 


RATING  SCALE 


Very  Poor 

Poor 

Marginal 

Acceptable 

Good 

Very  Good 

Superior 

1 

2 

3 

4 

5 

6 

7 
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Rating  Scale 


The  following  numeric  rating  scale  is  used  to  assess  the  level  of 
behavior  that  crews  exhibit  for  each  basic  quality  shown  on  the 
Aircrew  Coordination  Evaluation  (ACE)  Checklist  and  at  the  bottom 
of  the  Aircrew  Coordination  Training  Grade  Slip.  Each  basic 
quality  is  rated  using  a  seven-point  scale  with  values  ranging 
from  1  (very  poor)  to  7  (superior) : 


Very 

Poor 

Poor 

Marginal 

Accept¬ 

able 

Good 

Very 

Good 

Superior 

1 

2 

3 

4 

5 

6 

7 

Rating 

Guidelines 

Written  descriptions  of  the  types  of  behaviors  and  levels  of 
performance  are  shown  for  rating  values  1,  4,  and  7.  These 
descriptions  serve  as  behavioral  "anchors"  and  are  designed  to 
assist  evaluators  in  determining  how  well  a  crew  performs  on  each 
basic  quality  in  relation  to  a  well-defined  set  of  behaviors. 
Evaluators  should  use  the  "anchors"  as  the  standard  for  making 
ratings — avoid  comparing  one  crew's  performance  with  that  of 
another  crew's;  rate  a  crew's  performance  in  relation  to  the 
"anchors."  To  ensure  reliable  ratings,  continue  to  refer  to  the 
anchors  when  making  rating  responses  until  completely  confident 
and  understand  fully  how  to  rate  each  basic  quality. 

In  completing  a  basic  quality  rating,  evaluators  should  decide 
whether  the  behaviors  observed  fall  into  the  low  end  of  the  basic 
quality  range  (values  1  or  2),  the  middle  of  the  range  (values  3, 
4,  or  5),  or  the  high  end  of  the  range  (values  6  or  7) .  Once  the 
general  range  of  response  is  selected,  use  the  anchors  to  help 
select  the  final  rating  value.  For  example,  if  a  crew  did  an 
adequate  job  of  pre-mission  planning  and  rehearsal,  the  rating 
would  come  from  the  middle  of  the  range  (3,  4,  or  5) .  After 
determining  this,  review  the  behavioral  description  (anchor) 
associated  with  value  4  to  determine  if  crew  performance 
resembled  this  description  (4  value),  was  somewhat  less  than  this 
description  (3  value),  or  was  a  little  better  than  this 
description  (5  value) .  Use  the  end-point  anchors  similarly  to 
help  determine  ratings  that  fall  near  the  ends  of  the  scale. 

Army  aviation  crews  that  have  little  or  no  training  in  aircrew 
coordination  techniques  will  score  most  frequently  in  the  lower 
half  of  the  scale.  Most  other  crews,  however,  will  fall  into 
the  middle  area  of  the  scale.  Keep  in  mind  that  although  Army 
aviators  have  well  developed  basic  flying  skills,  as  a  group, 
their  aircrew  coordination  skills  will  be  much  like  the 
rest  of  the  population.  A  few  crews  will  have  strong 
coordination  and  communication  skills,  a  few  will  have  weak 
skills,  and  a  significant  number  will  have  moderate  skills. 
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Aircrew  Coordination  Basic  Qualities  and  Behavioral  Anchors 


BASIC  QUALITY  1.  Establish  and  maintain  flight  team  leadership 

and  crew  climate  (Crew  Climate) 


Explanation: 

This  rating  assesses  the  quality  of  relationships  among  the  crew 
and  the  overall  climate  of  the  flight  deck.  Aircrews  are  teams 
with  a  designated  leader  and  clear  lines  of  authority  and 
responsibility.  The  pilot-in-command  sets  the  tone  of  the  crew 
and  maintains  the  working  environment.  Effective  leaders  use 
their  authority  but  do  not  operate  without  the  participation  of 
other  crewmembers.  When  crewmembers  disagree  on  a  course  of 
action,  rate  the  crew's  effectiveness  in  resolving  the 
disagreement.  Note:  Traditional  leadership  centralizes 
leadership  in  the  leader  with  followers  fully  dependent  on  the 
leader.  Functional  leadership  assigns  leadership  and 
followership  roles  as  the  situation  evolves.  Flight  team 
leadership  recognizes  the  impact  of  leadership  style  on  the 
working  environment.  Regardless  of  leadership  style,  the  pilot- 
in-command  retains  final  decision  and  direction  authority. 

Superior  Rating  (7) 

The  crewmembers  have  very  good  interpersonal  relationships.  They 
respect  each  others'  skills  and  appear  to  enjoy  being  with  each 
other.  The  climate  is  very  open;  crewmembers  freely  talk  and  ask 
questions.  Crewmembers  encourage  the  individual  with  the  most 
information  about  the  situation-at-hand  to  participate.  There  is 
a  genuine  concern  for  good  working  relationships.  No  degrading 
comments  or  negative  voice  tones  are  used  in  interactions. 
Disagreements  are  perceived  as  a  normal  part  of  crew 
interactions,  and  the  crew  directly  confronts  the  issues  over 
which  the  disagreement  began.  Arguments  or  disagreements  focus 
on  behaviors  or  solutions  rather  than  on  personalities.  Each 
crewmember  carefully  listens  to  others'  comments.  Senior 
crewmembers  accept  challenges  from  junior  crewmembers. 

Alternative  solutions  are  explored.  The  solution  produced  is  a 
"win-win"  situation  in  which  all  crewmembers'  opinions  are 
considered.  The  crewmembers  have  no  hard  feelings  at  the 
conclusion  of  the  incident. 


Acceptable  Rating  (4) 

The  crewmembers  have  sound  interpersonal  relationships  and  seem 
to  respect  each  others'  skills.  The  climate  is  an  open  one,  and 
crewmembers  are  free  to  talk  and  ask  mission  questions. 

Regardless  of  rank  or  duty  position,  the  individual  with  the  most 
information  about  the  situation-at-hand  is  allowed  to 
participate.  When  disagreements  arise,  the  crew  directly 
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confronts  the  issues  over  which  the  disagreements  began.  The 
primary  focus  is  on  behaviors  or  solutions,  and  no  personal 
attacks  are  made  in  the  heat  of  discussion.  The  solution  is 
generally  seen  as  reasonable.  Problem  resolution  ends  on  a 
positive  note  with  very  little  hostility  or  grumbling  among 
crewmembers.  Mutual  respect  is  clearly  intact. 


Very  Poor  Rating  (1) 

Crew  interactions  are  often  awkward  and  uncomfortable.  The 
crewmembers  do  not  appear  to  like  or  respect  each  other. 
Crewmembers  may  be  curt  and  impolite  to  each  other.  Requirements 
for  assistance  are  made  as  commands  rather  than  as  requests  for 
support.  When  disagreements  arise,  the  crew  fails  to  directly 
confront  the  issues.  Personal  attacks  may  arise.  Senior 
crewmembers  are  resistant  to  recommendations  from  junior 
crewmembers.  Crewmembers  do  not  explore  the  range  of  possible 
solutions.  They  may  shout  and  argue  without  finding  a  solution. 
One  or  more  crewmembers  may  retreat  and  say  nothing  at  all.  A 
"win-lose"  situation  develops  in  which  one  crewmember  is  shown  to 
be  right  and  the  other  to  be  wrong.  The  crewmembers  show  little 
respect  to  one  another  except  for  deferring  to  formal  rank. 


BASIC  QUALITY  2.  Pre-mtssion  planning  and  rehearsal 

accomplished  (Plan  Rehearsal) 


Explanation: 

This  rating  assesses  the  pre-mission  planning  and  rehearsal 
activities  that  the  crew  performs  upon  receiving  a  mission  order. 
Time  available  determines  whether  pre-mission  planning  and 
rehearsal  is  completed  prior  to  the  flight  or  in  the  cockpit. 
During  this  period  crews — 

•Clarify  the  mission  order  and  the  commander's  intent 
•Assign  actions,  duties,  and  mission  responsibilities 
•Collect  information  (intelligence,  communications, 
weather,  flight  planning)  and  develop  the  plan 
•Conduct  crew  briefing  to  review  and  discuss  the  plan 
•Identify  potential  problem  areas  and  courses  of  action 
•Assess  risks 

•Visualize  and  rehearse  the  mission 

Although  the  pilot-in-command  is  responsible  for  leading  this 
activity,  evaluate  the  extent  and  manner  in  which  the  entire  crew 
participates.  Also,  consider  the  time  constraints  on  the  crew.  If 
there  is  insufficient  time  to  conduct  comprehensive  planning  and 
rehearsal,  evaluate  the  crew  on  their  planning  and  rehearsal  of 
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the  most  critical  segments  of  the  mission.  That  is,  either  prior 
to  the  flight  or  in  the  cockpit,  did  the  crew  address  the  most 
important  issues  given  the  time  available?  Note:  The 
relationship  among  crew  members  should  be  observed  during  this 
period  but  the  crew  climate  evaluation  should  be  made  on  rating 
basic  quality  1,  Flight  Team  Leadership  and  Crew  Climate. 

Exampl es : 

•UH-60  Task  2078  and  AH-64  Task  1033,  Perform  terrain  flight 
mission  planning:  The  crew  will  analyze  the  mission  in  terms  of 
METT-T  and  plan  the  flight  as  directed  by  the  PC.  The  crew  will 
rehearse  important  aspects  of  the  mission. 

•UH-60  and  AH-64  Task  1000,  Conduct  crew  mission  briefing: 

Aircrew  collectively  visualizes  and  rehearses  expected  and 
unexpected  events  from  takeoff  to  tie-down;  all  factors  of  the 
flight;  and  actions,  duties,  and  responsibilities  of  each 
crewmember. 

•AH-64  and  UH-60  Task  1068,  Perform  or  describe  emergency 
procedures:  PC  will  include  in  the  crew  briefing  the  general 
approach  to  all  emergency  procedures  requiring  immediate  action. 

Superior  Rating  (7) 

The  entire  crew  discusses  a  detailed  description  of  the  mission 
and  the  commander's  intent.  All  actions,  duties,  and  mission 
responsibilities  are  partitioned  and  clearly  assigned  to  specific 
individuals.  The  crew  acquires  new  and  updated  information  and 
uses  it  to  develop  the  mission  plan  from  the  aircrew  mission 
briefing.  Questions  and  discussion  about  the  mission,  commander's 
intent,  and  specific  responsibilities  are  encouraged.  Potential 
problems  are  noted  and  discussed  in  detail.  Courses  of  action  and 
individual  responsibilities  are  established  in  the  event  that 
potential  problems  actually  occur.  All  crewmembers  speak  out  and 
acknowledge  an  understanding  of  the  operational  risks  in  the 
mission  plan.  The  pi lot- in- command  leads  the  crew  in  mentally 
rehearsing  the  entire  mission  by  visualizing  and  talking  the  crew 
through  potential  problems  and  contingencies.  Crewmembers 
acknowledge  understanding  their  assigned  responsibilities  and 
cues  for  actions.  The  tone  of  the  interaction  is  friendly  and 
professional . 


Acceptable  Rating  (4) 

A  brief  description  of  the  mission  is  provided  to  the  entire 
crew.  The  mission  responsibilities  are  partitioned  and  assigned 
to  specific  individuals.  Actions  are  taken  to  update  current 
information  that  adds  to  the  aircrew  mission  briefing  and  helps 
develop  the  mission  plan.  One  or  more  crewmembers  make  comments 
during  the  course  of  developing  the  mission  plan.  Potential 
mission  problems  are  only  briefly  discussed.  There  is  adequate 
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preparation  for  contingencies.  Crewmembers  briefly  discuss  the 
operational  risks  in  the  mission  plan.  Mental  rehearsal  is 
initiated  by  the  pilot-in-command  or  another  crewmember  who  talks 
through  potential  problems  or  contingencies  for  one  or  more 
mission  segments.  Some  discussion  takes  place  to  clarify 
responsibilities  in  the  event  of  unexpected  problems  or 
contingencies.  The  tone  of  the  interaction  is  generally  friendly 
and  businesslike. 


Very  Poor  Rating  (1) 

The  pilot-in-command  briefs  the  mission  with  little  or  no 
attendant  explanation.  There  is  little  or  no  discussion  of 
responsibilities  or  their  assignments  to  specific  crewmembers. 

The  pi lot- in- command  develops  the  mission  plan  from  the  aircrew 
mission  briefing  and  current  information.  Crewmembers  tend  not  to 
ask  questions  about  the  mission.  If  asked,  questions  tend  to  be 
cut  off,  only  briefly  addressed,  or  ignored  by  the  other  crew¬ 
members.  Little  or  no  mention  is  given  to  potential  problems  or 
complications.  No  crewmember  says  anything  about  operational 
risks  or  weaknesses  in  the  plan.  Any  suggestion  to  talk  through  a 
potential  problem  or  mentally  rehearse  responsibilities  is 
rejected  as  unnecessary.  The  tone  of  the  interaction  is 
business-like,  abrupt,  and  impersonal. 


BASIC  QUALITY  3,  Application  of  appropriate  deci-making 

techniques  (Decision  Tech) 


Explanation: 

This  rating  evaluates  the  manner  and  quality  of  the  crew's 
problem  solving  and  decision  making  performance  throughout  the 
planning  and  execution  of  the  mission.  Factors  to  consider  in 
making  this  evaluation  include  (1)  information  available  to  the 
crewmembers,  (2)  time  urgency  of  the  decision,  (3)  objectivity 
reflected  in  the  decision  process,  and  (4)  level  of  involvement 
and  information  exchange  among  the  crewmembers.  The  time 
critical  demands  of  tactical  flying  require  many  decisions  to  be 
made  on  an  automatic,  pattern-recognition  basis  with  only  a 
minimum  level  of  information  exchange.  However,  when  adequate 
time  and  information  are  available,  crewmembers  are  expected  to 
engage  in  a  more  deliberate  and  interactive  style  of  decision 
making.  The  evaluation  of  crew  decision  making  performance 
should  ask  the  following  questions:  (1)  Did  the  crew  use  all  of 
the  available  information?  (2)  Was  the  level  of  information 
exchange  among  crewmembers  appropriate  for  the  time  available? 

(3)  Was  the  type  of  decision  process  (deliberate  versus 
automatic)  appropriate  for  the  time  available? 
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Exampl es : 

•UH-60  and  AH-64  Task  2044,  Perform  actions  on  contact:  Crew  will 
discuss  options  for  developing  the  situation,  then  choose  a 
course  of  action  that  supports  the  intent  of  the  unit  commander's 
directives . 

•AH-64  and  UH-60  Task  2083,  Negotiate  wire  obstacles:  Crew  will 
discuss  the  characteristics  of  the  wires  ...  to  determine  the 
method  of  crossing. 

Superior  Rating  (7) 

Crew  decision  making  consistently  reflects  proper  attention  to 
available  information  throughout  mission  planning  and  execution. 
The  level  of  crew  participation  and  deliberate  analysis  of 
options  is  appropriate  for  the  decision  time  available.  Resulting 
decisions  are  timely  and  appropriate  given  the  time  urgency  and 
level  of  information  available  in  each  situation.  Crewmembers  do 
not  exhibit  any  of  the  known  hazardous  thought  patterns  (e.g., 
anti-authority,  impulsivity,  machoism,  invulnerability, 
resignation,  get-home-itis,  overconfidence  in  other  aviator)  and 
appear  motivated  to  seek  the  most  mission  effective  and  safe 
decision  in  each  situation.  The  crew  decides  and  implements  a 
course  of  action  before  the  situation  jeopardizes  crew 
performance  or  mission  accomplishment. 


Acceptable  Rating  (4) 

Crew  decisions  occasionally  reflect  inadequate  sharing  or  use  of 
available  information.  On  limited  occasions,  crewmembers  dwell 
excessively  on  some  issues  while  neglecting  more  time  urgent 
requirements.  Most  decisions  are  timely,  but  crew  performance 
begins  to  show  signs  of  self-induced  stress.  Most  decisions  are 
appropriate  for  the  situation,  with  the  crew  occasionally 
overlooking  one  or  more  factors  or  options.  Crewmembers 
occasionally  fail  to  recognize  or  exploit  opportunities  for 
additional  planning  or  rehearsal,  substituting  instead  ad  hoc 
strategies  or  plans.  Crewmembers  do  not  exhibit  any  of  the  known 
hazardous  thought  patterns.  The  situation  may  worsen,  without 
seriously  degrading  mission  accomplishment,  before  the  crew 
decides  and  implements  a  course  of  action. 


Very  Poor  Rating  (1) 

Crew  performance  (both  pre-flight  and  in-flight)  reflects  an 
inflexible  style  of  decision  making  (either  deliberate  or 
automatic)  regardless  of  time  urgency.  Crewmembers  may  engage  in 
excessive  deliberation,  overlook  the  relative  time  urgency  of 
competing  decision  requirements,  or  fall  victim  to  inappropriate 
mind  sets.  As  a  result,  decisions  frequently  lack  timeliness, 
ignore  important  factors,  or  appear  out  of  context.  Information 
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exchange  and  crewmember  interaction  is  minimal,  with  the  result 
that  critical  input  is  ignored  or  not  sought.  Crewmembers  may 
display  one  or  more  of  the  known  hazardous  thought  patterns 
(e.g.,  machoism,  anti-authority,  get-home-itis) .  The  crew  may  be 
unable  to  decide  or  implement  a  course  of  action  before  a 
situation  becomes  critical. 


BASIC  QUALITY  4.  Prioritize  actions  and  distribute  vorkload 

(Workload) 


Explanation: 

This  is  a  rating  of  the  effectiveness  of  time  and  work 
management.  Rate  the  extent  to  which  the  crew  as  a  team  avoids 
being  distracted  from  essential  activities,  distributes  workload, 
and  avoids  individual  crewmember  overload. 

Examples: 

•AH-64  and  UH-60  Task  1080,  Perform  procedures  for  two-way  radio 
failure:  P*  will  remain  focused  outside  the  aircraft  or  inside 
the  cockpit  on  the  instruments,  as  appropriate.  He  will  not 
participate  in  troubleshooting  the  malfunction. 

•UH-60  Task  2079  and  AH-64  Task  1064,  Perform  terrain  flight 
navigation:  P  will  focus  his  attention  primarily  inside  the 
cockpit;  however,  as  workload  permits,  he  will  assist  in  clearing 
the  aircraft  and  provide  adequate  warning  of  traffic  and 
obstacles. 


Superior  Rating  (7) 

Virtually  all  distractions  are  avoided.  Each  crewmember 
understands  precisely  what  information  is  relevant  to  the  mission 
and  what  information  is  simply  a  distraction.  If  a  crewmember 
becomes  mildly  distracted,  other  crewmembers  remind  him  to  focus 
on  the  mission  task.  Non-critical  duties  are  prioritized  and 
delayed  until  low  workload  periods  or  post-flight  periods. 
Crewmembers  are  aware  of  workload  build  ups  on  others  and 
readjust  workload  by  assuming  emerging,  unassigned  tasks 
appropriate  for  their  duty  station.  Overloads  do  not  occur.  The 
crew's  planning  horizon  is  always  "ahead  of  the  aircraft." 


Acceptable  Rating  (4) 

Most  distractions  are  avoided.  The  crew  performs  well  in 
deciding  what  information  and  activities  are  essential  to  the 
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mission.  Most  non-essential  information  is  discarded  or  ignored. 
Non-critical  duties  are  prioritized  and  delayed  until  low 
workload  periods  or  post-flight  periods.  Crewmembers  are  aware 
of  individual  crewmember  workloads  during  each  phase  of  the 
mission.  When  an  individual  crewmember  appears  to  be  overloaded, 
other  crewmembers  take  on  part  of  the  workload.  The  crew  is 
always  "in  sync  with  the  aircraft." 


Very  Poor  Rating  (1) 

The  crew  is  easily  distracted.  The  crew  is  unable  or  unwilling 
to  decide  what  is  important  and  relevant  to  the  immediate 
mission.  There  is  little  prioritizing  of  duties  or  actions. 

Time  and  energy  may  be  wasted  on  low  priority  tasks.  Risks  to 
crew  safety  may  occur  as  the  crew  focuses  on  minor  tasks  while 
critical  tasks  requiring  immediate  attention  go  unattended, 

(e.g.,  setting  a  radio  frequency  when  attention  should  be  focused 
on  clearing  an  obstacle.).  Neither  the  overloaded  party  nor 
other  crewmembers  takes  voluntary  actions  to  eliminate  an 
overload  condition.  The  crew  makes  little  or  no  effort  to 
redistribute  task  responsibilities  as  mission  changes  occur  and 
new  tasks  arise.  Individual  crewmembers  experience  workload 
overloads.  The  crew's  planning  horizon  is  sometimes  "behind  the 
aircraft . " 


BASIC  QUALITY  5.  MAnsigement  of  unexpected  events  (Unexp  Events) 


Explanation: 

This  rating  evaluates  the  crew's  performance  under  unusual 
circumstances  that  may  involve  high  levels  of  stress.  This 
judgement  includes  the  integration  of  technical  and  managerial 
aspects  of  contending  with  the  situation.  Note:  Enter  the 
abnormal  or  emergency  situation  in  the  Aircrew  Coordination 
Training  Grade  Slip  (some  emergency  procedure  ATM  tasks  are 
preprinted)  and  grade  it  the  same  as  any  task. 

Examples: 

•AH-64  and  UH-60  Task  2008,  Perform  evasive  maneuvers:  The  most 
important  consideration  in  an  emergency  is  aircraft  control — 
first  assess  aircraft  controllability,  check  systems  indicators, 
take  evasive  action. 

•UH-60  Task  1068,  Perform  or  describe  emergency  procedures:  CE 
will  keep  communications  to  a  minimum  to  allow  the  P*  or  P  to 
attempt  communications  outside  the  aircraft. 
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Superior  Rating  (7) 

The  crew  remains  calm  during  the  situation.  Each  crewmember 
seeks  to  understand  the  problem  and  provides  the  pilot-in-command 
with  essential  information.  Each  crewmember  immediately  takes  on 
particular  workload  responsibilities  based  on  prior  discussions 
and  rehearsal  of  potential  problems  and  contingencies .  The  crew 
effectively  communicates  its  actions  and  results  to  others  and 
provides  feedback  to  ensure  complete  coordination  of  efforts. 

Each  crewmember  handles  his  own  responsibilities  and  seeks  to 
support  the  crewmember  with  the  greatest  workload.  The  crew 
rapidly  imposes  the  maximum  amount  of  control  possible  over  the 
situation  given  the  available  time  and  internal  and  external 
resources.  A  high  level  of  situation  awareness  is  maintained 
throughout  the  event. 


Acceptable  Rating  (4) 

The  crew  responds  to  the  problem  and  the  pilot-in-command's 
requests  for  information  but  does  not  overreact.  The 
pilot-in-command's  requests  for  information  are  met  by  feedback 
from  the  crew.  The  crew  takes  actions  to  reduce  the 
pilot-in-command's  work  overload  and  provides  information  even  if 
it  is  not  specifically  requested.  The  pilot  and  crew  make  good 
use  of  available  resources.  The  crew  is  intense  but  not 
flustered  by  the  situation.  Adequate  situation  awareness  is 
maintained  throughout  the  event. 


Very  Poor  Rating  (1) 

The  crew  becomes  disorganized  and  flustered.  The  pilot-in¬ 
command's  requests  for  information  elicit  inadequate  responses. 
Crewmembers  may  focus  on  the  wrong  issues,  thus  delaying  correct 
diagnosis  of  the  problem.  The  crew  focuses  on  only  one  solution 
to  an  event,  does  not  consider  other  plausible  alternatives,  or 
chooses  an  inappropriate  solution.  Lack  of  coordinated  actions 
adds  to  the  confusion.  The  pilot  and  crewmembers  make  poor  use 
of  available  resources  to  resolve  the  problem.  Situation 
awareness  appears  to  decay  during  the  situation. 


BASIC  QUALITY  6.  Statements  and  directives  clear,  timely, 

relevant,  complete,  and  verified  (Info  Xfer) 


Explanation: 

Rate  the  completeness,  timeliness,  and  quality  of  information 
transfer.  Carefully  consider  the  crew's  feedback  techniques  to 
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verify  information  transfer.  In  particular,  evaluate  the  quality 
of  instructions  and  statements  associated  with  navigation 
activities,  obstacle  clearing  activities,  and  instrument 
readouts , 

Exampl es : 

•AH-64  Task  1015,  Perform  ground  taxi:  The  P  will  announce 
"Blocking"  to  acknowledge  the  P* ' s  announcement  "Braking" . 

•UH-60  Task  2079,  Perform  terrain  flight  navigation:  The  P*  will 
acknowledge  commands  issued  by  the  P  for  heading  and  airspeed 
changes . 

Superior  Rating  (7) 

Crewmembers  communicate  frequently.  Both  senders  and  receivers 
use  standard  terminology  for  nearly  all  communications.  Senders 
almost  always  provide  clear,  concise  information.  Receivers 
acknowledge  nearly  all  messages  in  sufficient  detail  so  that  the 
sender  can  verify  that  the  receiver  understands  the  message. 
Receivers  ask  for  clarification  when  they  do  not  understand. 
Senders  pursue  feedback  when  no  response  is  forthcoming.  Whenever 
a  workload  shift  or  task  responsibility  transfer  occurs,  the 
change  is  communicated  and  acknowledged  by  the  crew.  All 
navigation,  obstacle  clearing,  and  "inside"  or  "outside"  the 
cockpit  information  is  stated,  acknowledged,  and  updated. 


Acceptcible  Rating  (4) 

Crewmembers  communicate  about  the  mission  as  required.  Standard 
terminology  is  usually  used.  Receivers  acknowledge  most 
messages.  Receivers  ask  questions  when  they  do  not  understand. 
Senders  usually  pursue  feedback  when  no  response  is  forthcoming. 
Crewmembers  are  appraised  of  changes  to  responsibilities  during 
the  flight..  "Inside"  and  "outside"  the  cockpit  duties  are 
specified  and  communicated  to  others. 


Very  Poor  Rating  (1) 

Crewmembers  may  fail  to  make  statements  regarding  critical 
information.  Non-standard  terminology  is  used  or  standard 
terminology  is  used  inappropriately.  Sender  messages  may  be 
inappropriately  delayed  or  irregular  and  may  be  confusing. 
Receivers  usually  do  not  verbally  acknowledge  the  receipt  of 
messages.  Receivers  do  not  ask  questions.  Senders  do  not  pursue 
feedback  when  no  response  is  forthcoming.  Changes  in 
responsibilities  during  the  mission  are  often  not  communicated 
and  may  result  in  confusion  over  who  has  a  task  responsibility. 
Navigation  instructions  and  obstacle  location  information  may  be 
incomplete  or  confusing.  At  times,  "inside"  or  "outside"  the 
cockpit  responsibilities  are  not  clearly  communicated. 
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BASIC  QUALITY  7.  Maintenance  of  mission  situation  awareness 

{Sit  Aware) 


Explanation: 

This  rating  assesses  the  extent  to  which  crewmembers  keep  each 
other  informed  on  the  status  of  the  aircraft  and  mission 
accomplishment.  This  information  reporting  helps  maintain  a  high 
level  of  situation  awareness  among  the  flight  crew.  Information 
reported  includes: 

•Aircraft  position  and  orientation 
•Equipment  status 
•Personnel  status 

•Environment  and  battlefield  conditions 
•Changes  to  mission  objectives 

Crew-wide  situation  awareness  is  an  essential  element  of  safe 
flying  and  effective  crew  performance. 

Exampl es : 

•UH-60  Task  2009,  Perform  multi-aircraft  operations:  P  and  CE 
will  provide  adequate  warning  to  avoid  traffic  or  obstacles. 

•AH-64  Task  2008,  Perform  evasive  maneuvers:  When  engaged  by  the 
enemy,  crew  will  announce  the  nature  and  direction  of  the  threat. 

Superior  Rating  (7) 

Crewmembers  routinely  provide  each  other  with  updates  on  the 
status  of  the  elements  of  situation  awareness  and  the  status  of 
the  mission.  Crewmembers  anticipate  the  situation  awareness 
needs  of  others  and  request  needed  information  when  it  is  not 
forthcoming.  Crewmembers  are  aware  of  each  others'  mental  and 
physical  states  and  are  not  hesitant  to  alert  others  to  personal 
problems  that  could  undermine  effective  performance.  Personnel 
status  is  voluntarily  shared  without  fear  of  sanctions.  All 
changes  in  the  elements  of  situation  awareness  are  verbalized  and 
acknowledged.  Crewmembers  alert  other  crewmembers-  to  the 
presence  of  obstacles. 


Acceptable  Rating  (4) 

Crewmembers  usually  provide  updates  on  the  status  of  most  of  the 
elements  of  situation  awareness  and  the  status  of  the  mission. 
Changes  to  the  situation  awareness  elements  are  verbalized. 
Obvious  changes  in  personnel  status  are  noted  and  acknowledged 
without  fear  of  sanctions. 
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Very  Poor  Rating  (1) 

Crewmembers  do  not  routinely  provide  updates  on  the  status  of  the 
aircraft  or  the  status  of  the  mission.  Generally,  updates  are 
provided  only  on  request;  they  are  not  made  voluntarily. 

Personnel  problems  such  as  fatigue  or  lack  of  attention  are  not 
mentioned. 


BASIC  QUALITY  8.  Decisions  and  actions  communicated  and 

acknovledged  (Comm/  Ack) 


Explanation: 

Rate  the  extent  to  which  decisions  and  actions  are  actually  made 
and  announced  to  the  crewmembers  after  input  is  solicited  from 
them.  Crewmembers  should  respond  verbally  or  with  the 
appropriate  adjustment  to  their  behaviors,  actions,  or  control 
inputs  to  clearly  indicate  that  they  understand  when  a  decision 
has  been  made  and  what  it  is.  Failure  to  do  so  may  confuse  crews 
and  lead  to  uncoordinated  operation.  Note:  Due  to  time 
constraints  in  certain  situations,  there  is  often  little  or  no 
time  for  crews  to  make  inputs  to  a  decision.  In  such  cases, 
raters  should  focus  on  the  extent  to  which  decisions  are 
acknowledged  verbally  or  through  coordinated,  pre-planned  action. 

Ex amp 1 es : 

•UH-60  Task  2086,  Perform  masking  and  unmasking:  P*  will  announce 
his  intent  to  unmask.  The  P  and  CE  will  acknowledge  that  they 
are  prepared  to  execute  the  maneuver. 

•AH-64  Task  1038,  Perform  terrain  flight  approach:  P*  will 
announce  intention  of  a  go-around  .  .  .  whether  approach  will 
terminate  to  a  hover  or  to  the  ground.  P  will  acknowledge  use  of 
manual  stabilator  or  any  intent  to  deviate  from  the  approach. 

Superior  Rating  (7) 

The  pilot-in-command  states  decisions  and  actions  and,  time 
permitting,  explains  the  reasons  and  intent.  Crewmembers 
acknowledge  the  decisions  with  a  clear  verbal  response  and  ask 
questions  to  clarify  any  confusion.  The  pilot-in-command  answers 
all  questions  in  a  positive,  straight-forward  manner. 

Crewmembers  keep  the  pilot-in-command  informed  of  the  results  of 
their  activities  and  changing  responsibilities — especially  visual 
area  of  responsibility  or  task  focus.  The  crew  clearly 
acknowledges  results  of  actions,  or  changes,  and  then  states  its 
intended  adjustments  based  on  the  information  provided.  If 
crewmembers  do  not  acknowledge  or  adjust,  the  pilot-in-command 
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requests  acknowledgement.  Crewmembers  are  particularly  attentive 
to  the  communication  of  workload  responsibilities.  When  assuming 
control  of  the  aircraft  or  making  control  inputs,  notification  is 
always  given  and  acknowledgement  received. 


Acceptable  Rating  (4) 

The  pilot-in-command  states  decisions  and  actions  along  with, 
time  permitting,  a  brief  explanation  of  the  reasons  and  informs 
the  crew  of  the  adjustments  they  are  expected  to  make.  The  crew 
acknowledges  its  awareness  of  the  decisions  and  directions. 
Crewmembers  may  ask  questions  to  clarify  confusion.  The  pilot 
answers  questions  clearly  and  quickly  and  the  crew  adjusts  to  the 
new  situation.  When  assuming  control  of  the  aircraft  or  making 
control  inputs,  notification  is  given  and  acknowledged. 


Very  Poor  Rating  (1) 

Decisions  and  actions  of  a  crewmember  are  often  not  passed  on  to 
the  crew.  The  pilot-in-command  takes  unilateral  action  and  does 
not  explain  or  inform  the  crew  of  his  intended  purpose.  The  crew 
is  often  not  aware  that  a  decision  has  been  made.  The  crew 
infrequently  asks  questions  for  clarification.  The 
pi lot- in- command  may  not  acknowledge  or  respond  to  questions.  The 
crew  may  not  know  how  to  react  to  changed  circumstances. 
Crewmembers  are  often  unsure  what  responsibilities  have  been 
assigned  to  them.  Crewmembers  may  take  uncoordinated  actions 
without  stating  intentions  or  results.  Two  pilots  may  attempt  to 
simultaneously  take  control  of  the  aircraft  when  flight  control 
authority  is  unclear. 


BASIC  QUALITY  9.  Supporting  information  and  actions  sought  from 

crew  (Info  Sought) 


Explanation: 

This  is  a  rating  of  the  extent  to  which  crewmembers,  usually  the 
pilot-in-command,  seek  support  information  and  support  actions 
from  the  crew.  Evaluate  the  degree  to  which  crewmembers  raise 
questions  during  the  flight  regarding  plans,  revisions  to  plans, 
actions  to  be  taken,  and  the  status  of  key  mission  information. 
Note:  The  extent  to  which  crewmembers  maintain  situational 

awareness  and  contribute  to  decision  making  should  be  observed 
here  but  evaluated  on  basic  qualities  7  and  4  respectively. 

Examples: 
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•UH-60  Task  1032,  Perform  slope  operations:  P*  will  request 
assistance  in  setting  the  brakes. 

•AH-64  Task  2044,  Perform  actions  on  contact:  The  crew  will 
discuss  options  for  developing  the  situation. 


Superior  Rating  (7) 

During  the  flight,  crewmembers  raise  questions  on  plans  or 
changes  to  plans  and  actions.  Virtually  all  of  these  inquiries 
surface  information  that  contributes  to  the  mission  decision 
making  process.  When  the  pilot-in-command  realizes  that  a 
decision  must  be  made  during  the  flight,  for  which  there  is  no 
clear  standardized  answer,  he  immediately  alerts  the  crew  to  the 
situation  and  seeks  suggestions  on  possible  solutions  and 
important  information  to  consider.  The  pilot-in-command  is  open 
to  all  suggestions.  Crewmembers  respond  to  these  inquiries  with 
sound,  task-focused  discussions  and  clear  answers  that  are 
provided  in  a  timely  manner.  Crewmembers'  inquiries  are  never 
ignored.  All  crewmembers  encourage  such  questioning.  When  the 
pilot-in-command  asks  for  assistance  with  actions  he  clearly 
states  what  assistance  is  required.  He  provides  quick,  clear 
feedback  if  the  crewmember  response  is  not  what  he  expects.  He 
asks  for  assistance  before  becoming  overloaded. 


Acceptcible  Rating  (4) 

During  the  flight,  crewmembers  occasionally  raise  questions  on 
plans  or  actions  when  they  are  unclear  on  decisions  being  made. 
Most  of  these  inquiries  provide  information  that  is  relevant  to 
the  mission  decision  making  process.  The  pilot  alerts  the  crew 
to  the  need  for  decision  input.  Crewmembers  usually  respond  to 
these  inquiries  with  brief  but  reasonable  answers.  Crewmembers' 
inquiries  are  encouraged  by  other  crewmembers  most  of  the  time. 
The  pilot-in-command  listens  to  suggestions  without  interruption 
or  criticism.  He  asks  for  clarification  as  necessary.  He  only 
asks  for  assistance  when  he  becomes  overloaded. 


Very  Poor  Rating  (1) 

During  the  flight,  crewmembers  almost  never  raise  questions  about 
plans,  actions,  or  changes  to  plans.  The  pilot-in-command  makes 
mission  decisions  without  seeking  inputs  from  other  crewmembers. 
The  pilot-in-command  does  not  alert  the  crew  that  a  decision  is 
required  or  is  being  made.  Decision  making  and  planning  are  done 
by  one  individual  with  little  or  no  discussion — an  observer  will 
have  difficulty  noting  this  quality  for  "very  poor"  crews  since 
it  is  hard  to  detect  individual  decision  making.  The  few 
inquiries  that  are  made  are  generally  ignored  or  abruptly 
answered.  Crewmembers  may  discourage  others  from  asking 
questions  by  the  tone  of  voice  they  use  or  by  failing  to  respond. 
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The  pilot-in-command  may  not  ask  for  crew  assistance  with  tasks 
even  when  he  is  overloaded  to  the  point  of  nearly  failing  to 
properly  execute  tasks. 


BASIC  QUALITY  10.  Crewnewber  actions  mutually  cross  monitored 

(Cross  Monitor) 


Expl ana ti on : 

This  rating  captures  the  extent  to  which  a  crew  uses  cross 
monitoring  as  a  mechanism  to  avoid  errors  and  improve  future 
performance.  Crewmembers  are  able  to  catch  each  other's  errors. 
Such  redundancy  is  particularly  important  when  crews  are  fatigued 
or  overly  focused  on  critical  task  elements,  and  thus  more  prone 
to  make  errors.  Included  in  this  rating  is  the  crew's  use  of 
aircraft  technical  manual  checklists  to  perform  required 
procedure  checks  and  procedures  (i.e.,  engine-start,  run-up, 
before-takeoff,  before-  and  after-landing,  shutdown  checks;  HIT 
and  emergency  procedures) .  Note:  This  quality  does  not  imply 
that  task  responsibilities  are  not  clearly  defined.  It  asks  the 
question  "To  what  extent  do  crewmembers  help  an  individual 
assigned  primary  responsibility  for  a  task  or  action  by  reviewing 
the  quality  of  that  individual's  task  execution  and  alerting  him 
to  any  mistake  noted?" 

Examples: 

•AH-64  Task  1094,  Identify  major  US  or  allied  equipment  and  major 
threat  equipment:  P*  or  P  will  announce  the  type  and  direction  of 
the  equipment  detected.  The  other  crewmember  will  confirm  the 
type  and  direction  of  the  equipment. 

•UH-60  task  1023,  Perform  fuel  management  procedures:  PC  will 
confirm  the  results  of  the  fuel  check. 

Superior  Rating  (7) 

Each  crewmember  is  concerned  that  all  tasks  are  properly  executed 
and  checks  both  his  tasks  and  those  of  others.  When  mistakes  are 
noted,  the  crewmember  making  the  error  is  quickly  informed  in  a 
concise  manner  without  excessive  formality.  The  mistake  maker 
accepts  this  review  and  feedback  as  a  normal  part  of  crew 
operations . 


Acceptable  Rating  (4) 

Crewmembers  often  check  each  other's  task  performance  for  errors. 
Mistake  makers  are  informed  and  make  the  needed  corrections.  Only 
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occasionally  are  mistake  makers  annoyed  at  being  checked  and 
corrected. 


Very  Poor  Rating  (1) 

Crewmembers  seldom,  if  ever,  check  each  other's  task  execution. 
Crewmembers  are  insulted  if  they  are  corrected  by  another 
crewmember . 


BASIC  QUALITY  11.,  Supporting^  information  and  actions  offered 

by  crev  (Info  Offered) 


Explanation: 

This  is  a  rating  of  the  extent  to  which  crewmembers  anticipate 
and  offer  support  information  and  support  actions  to  the  decision 
maker,  usually  the  pilot-in-command,  when  it  becomes  apparent 
that  a  decision  must  be  made  or  an  action  taken. 


Exampl es : 

•UH-60  Task  2016,  Perform  external  load  operations:  All 
crewmembers  will  assist  in  clearing  the  aircraft  and  will  provide 
adequate  warning  of  obstacles,  unusual  drift,  or  altitude 
changes . 

•UH-60  and  AH-64  Task  1081,  Perform  nonprecision  approach:  P  will 
call  out  the  approach  procedure  to  the  P* . 

Superior  Rating  (7) 

The  crew  recognizes  that  a  decision  must  be  made  and  offers 
suggestions  and  information  to  the  pilot-in-command.  The  crew 
checks  for  responses  that  indicate  understanding.  The 
information  is  repeated,  as  necessary,  to  ensure  that  the  pilot- 
in-command  understands  the  input.  Pilot-in-command  responses  can 
be  verbal  or  non-verbal  actions.  The  crew  seeks  information  and 
provides  it  to  support  decisions  and  actions.  The  crew 
frequently  offers  task  execution  support.  The  support  offered 
always  reflects  the  pilot- in-command ' s  task  needs.  Crews  axe 
quick  to  offer  support  during  particularly  difficult  tasks  such 
as  obstacle  clearing. 
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Acceptable  Rating  (4) 

The  crew  recognizes  that  a  decision  or  action  must  be  made  and 
offers  suggestions  and  information  to  the  pilot-in-command.  The 
crew  sometimes  offers  task  execution  support.  Crewmembers 
usually  offer  obstacle  clearing  support. 


Very  Poor  Rating  (1) 

The  crew  does  not  offer  suggestions  and  inputs  to  support 
decision  making  or  actions.  Moreover,  it  often  appears  that  the 
crew  does  not  even  realize  that  a  decision  is  being  made.  The 
crew  generally  does  not  offer  its  services  to  support  task 
execution  for  other  crewmembers.  Crewmembers  may  fail  to  offer 
obstacle  clearing  support. 


BASIC  QUALITY  12,  Advocacy  and  assertion  practiced 

(Advoc/Assert) 


Explanation: 

This  rating  evaluates  the  extent  to  which  crewmembers  advocate  a 
course  of  action  they  consider  best,  even  when  it  may  differ  with 
the  one  being  followed  or  proposed.  Note:  Except  under  extreme 
emergency  conditions  where  time  is  absolutely 

critical,  it  is  usually  in  the  crew's  best  interest  to  hear  the 
full  range  of  viewpoints  available. 

Exampl es : 

•UH-60  and  AH-64  Task  2083,  Negotiate  wire  obstacles:  Crew  will 
discuss  the  characteristics  of  the  wires  ...  to  determine  the 
method  of  crossing. 

•AH-64  Task  2044,  Perform  actions  on  contact:  Crew  will  discuss 
options  for  developing  the  situation. 

Superior  Rating  (7) 

Crewmembers  state  to  the  rest  of  the  crew  a  course  of  action  that 
they  consider  best.  They  clearly  explain  their  reasons  for 
believing  this  to  be  the  best  course.  Other  crewmembers  listen 
to  the  argument  before  presenting  any  criticism  or  proposing 
alternate  courses.  Discussions  focus  on  the  strengths  and 
weaknesses  of  the  proposed  course  of  action,  not  on  the 
personality  of  the  crewmember  who  proposed  the  action. 
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Crewmembers  call  the  crew's  attention  to  changes  in  the  situation 
and  provide  information  that  is  essential  to  the  proper  execution 
of  another  crewmember's  task.  Crewmembers  pursue  feedback  to 
ensure  that  their  views  are  heard  and  understood.  Other 
crewmembers  expect  such  open  comments  and  view  them  as  positive 
contributions  to  mission  performance. 


Acceptable  Rating  (4) 

Crewmembers  state  their  support  for  a  course  of  action  or  suggest 
improvements  to  other  proposed  actions.  Each  crewmember  makes  an 
effort  to  explain  his  position  and  convince  others  to  concur  with 
him  on  the  course  of  action  to  be  taken.  Other  crewmembers  may 
interrupt  with  their  views  and  alternatives.  Crewmembers  usually 
speak  out  when  they  recognize  a  departure  from  the  mission  plan 
or  standard  procedures  or  when  they  have  a  piece  of  information 
that  is  important  to  another's  task  execution.  Crewmembers  seek 
assurances  that  presented  information  has  been  received.  Other 
crewmembers  view  such  comments  as  constructive  and  not  as  a 
challenge  to  authority. 


Very  Poor  Rating  (1) 

The  crew  almost  never  suggests  a  course  of  action.  Crewmembers 
attempting  to  propose  a  course  of  action  may  be  cut-off  before 
they  can  propose  the  action  or  explain  the  rationale  for  that 
action.  Crewmembers  proposing  courses  of  action  may  receive 
personal  attacks.  The  crew  raises  few,  if  any  concerns. 
Crewmembers  may  even  fail  to  intervene  when  risks  such  as 
obstacles  or  poor  visibility  arise. 


BASIC  QUALITY  13,  Crev^level  after-action  reviews  accomplished 

(AAR) 


Explanation: 

This  rating  evaluates  the  extent  to  which  the  crew  reviews  and 
critiques  its  decisions  and  actions  during  or  following  a  mission 
segment,  during  low  workload  periods,  or  during  the  post  flight 
debrief.  Evaluate  the  crew  on  their  discussion  of  strengths  and 
weaknesses  (for  example,  what  was  done  wrong,  what  might  be  done 
better,  how  improvements  can  be  made,  and  what  was  done  very 
well)  in  flight  skills  and  aircrew  coordination. 

Superior  Rating  (7) 

The  entire  crew  reviews  and  critiques  its  decisions  and  actions 
throughout  the  mission,  including  the  pre-mission  planning  and 
rehearsal  process.  Crewmembers  review  factors  considered  in 
making  their  decisions,  identify  additional  options  or  factors. 
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including  ways  to  "buy  time, "  that  should  have  been  considered, 
and  discuss  different  methods  of  weighing  information  in  the 
decision  process.  All  discussions  focus  on  behaviors  and 
information  and  carefully  avoid  any  "finger-pointing"  tones.  The 
focus  is  clearly  on  education  and  understanding  to  improve 
individual  and  collective  performance. 


Acceptable  Rating  (4) 

Senior  crewmember (s)  review  and  critique  the  crew's  decisions  and 
actions  during  problematic  segments  of  the  mission.  They 
determine  the  major  mistakes  in  the  crew's  actions  or  decisions 
and  identify  remedial  actions  or  alternative  options  for  future 
missions.  Although  the  critiques  are  intended  to  educate  the 
crew  and  to  improve  their  performance  during  future  missions, 
they  may  include  some  accountability  for  unsatisfactory 
performance. 


Very  Poor  Rating  (1) 

The  crew  either  fails  to  review  and  critique  its  mission 
performance  or  if  a  critique  is  performed,  it  is  punitive  or 
accusatory.  That  is,  the  critique  is  conducted  primarily  to 
assign  blame  for  unsatisfactory  performance.  Little  effort  is 
made  to  identify  lessons  learned  or  to  suggest  constructive  ways 
to  improve  future  performance . 
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APPENDIX  B-2 


ACE  CHECKLIST  FREQUENCY  TABLES 

BQl  EstaJDlish  and  maintain  flight  team  leadership  and  crew  climate 
Pretraining  Results: 

Rating  Value  Frequency  Percent 


Poor 

2 

1 

6.3 

Marginal 

3 

3 

18.8 

Acceptable 

4 

8 

50.0 

Good 

5 

4 

25.0 

Total 

16 

100.0 

Mean 

3.938 

Std  dev 

.854 

Valid  cases 

Post- 

-training  Results: 

Rating 

Value 

Frequency 

Percent 

Acceptable 

4 

6 

37.5 

Good 

5 

6 

37.5 

Very  Good 

6 

1 

6.3 

Superior 

7 

3 

18,8 

Total 

16 

100.0 

Mean 

5.063 

Std  dev 

1.124 

Valid  cases 

BQ2  Premission  planning  and  rehearsal  accomplished 


Pre training  Results: 

Rating 

Poor 

Marginal 

Acceptable 

Good 


Mean  3.063 


Value  Frequency 


2  3 

3  10 

4  2 

5  1 

Total  16 


Std  dev 


Percent 

18.8 

62.5 

12.5 
6.3 

100.0 

Valid  cases 


Post-training  Results: 
Rating 


Poor 

Acceptable 

Good 

Very  Good 
Superior 


Value  Frequency  Percent 


2  1  6.3 

4  8  50.0 

5  3  18.8 

6  3  18.8 

7  1  6.3 


Total  16  100.0 


Mean 


4 . 625  Std  dev 


1.204  Valid  cases 


16 


16 


B~2-l 


BQ3  Selection  of  appropriate  decision  models 
Pre training  Results: 


Rating 

Value 

Frequency 

Percent 

Poor 

2 

1 

6.3 

Marginal 

3 

11 

68.8 

Acceptable 

4 

4 

25.0 

Total 

16 

100.0 

Mean  3.188 

Std  dev 

.544 

Valid  cases 

16 

Post-training  Results: 

Rating  Value 

!  Frequency  Percent 

Marginal 

3 

1 

6.3 

Acceptable 

4 

11 

68.8 

Good 

5 

3 

18.8 

Superior 

7 

1 

6.3 

Total 

16 

100.0 

Mean  4.313 

Std  dev 

,873 

Valid  cases 

16 

BQ4  Prioritize  actions  and  distribute  workload 


Pre training  Results: 

Rating 

Very  Poor 
Poor 

Marginal 

Acceptable 


Mean  2.938 


Value  Frequency 


1  1 

2  3 

3  8 

4  4 

Total  16 


Std  dev 


Percent 

6.3 

18.8 

50.0 

25.0 

100.0 

Valid  cases 


Post-training  Results: 


Rating  Value 

Marginal  3 

Acceptable  4 

Good  5 

Very  Good  6 

Superior  7 

Total 

Mean  4.438  Std  dev 


Frequency  Percent 

2  12.5 

9  56.3 

2  12.5 

2  12.5 

1  6.3 

16  100.0 

1.094  Valid  cases 


16 


16 


B-2-2  ' 


BQ5  Management  of  unexpected  events 
Pretraining  Results: 


Rating 

Value 

Frequency 

Percent 

Poor 

2 

5 

31.3 

Marginal 

3 

7 

43.8 

Acceptable 

4 

4 

25.0 

Total 

*  16 

100.0 

Mean 

2.938 

Std  dev 

.772 

Valid  cases 

Post- 

-training  Results: 

Rating 

Value 

Frequency 

Percent 

Marginal 

3 

5 

31.3 

Acceptable 

4 

6 

37.5 

Good 

5 

4 

25.0 

Superior 

7 

1 

6.3 

Total 

16 

100.0 

Mean 

4.125 

Std  dev 

1.088 

Valid  cases 

BQ6  Statements  and  directives  clear,  timely,  relevant,  coic^^lete,  and 
verified 


Pretraining  Results: 

Rating 

Poor 

Marginal 

Acceptable 

Good 


Mean  3.125 

Post-training  Results: 

Rating 

Marginal 

Acceptable 

Good 

Very  Good 
Superior 


Value  Frequency  Percent 


2 

3 

4 

5 

Total 
Std  dev 


4 

7 

4 

1 

16 

.885 


25.0 

43.8 

25.0 

6.3 

100.0 


Valid  cases 


16 


Value  Frequency  Percent 


Mean 


4.438 


3 

4 

5 

6 
7 

Total 
Std  dev 


2 

8 

4 

1 

1 

16 

1.031 


12.5 

50.0 

25.0 

6.3 

6.3 

100.0 


Valid  cases 


16 


B-2-3 


BQ7  Maintenance  of  mission  situational  awareness 


Pretraining  Results: 

Rating 

Poor 

Marginal 

Acceptable 

Mean  2.813 


Value  Frequency 


2  6 

3  7 

4  3 

Total  16 


Std  dev 


Percent 


37.5 

43.8 

18.8 

100.0 

Valid  cases 


Post- training  Results : 


Rating  Value 

Poor  2 

Marginal  3 

Acceptable  4 

Good  5 

Superior  7 

Total 

Mean  3.750  Std  dev 


Frequency  Percent 

2  12.5 

6  37.5 

4  25.0 

3  18.8 

1  6.3 

16  100.0 

1.291  Valid  cases 


16 


16 


BQd  Decisions  and  actions  communicated  and  acknowledged 
Pretraining  Results: 

Rating  Value  Frequency  Percent 


Poor 

Marginal 

Acceptable 


Mean  3.250 

Post-training  Results: 

Rating 

Poor 

Marginal 

Acceptable 

Good 

Very  Good 
Superior 


2 

3 

4 

Total 
Std  dev 


18.8 

37.5 

43.8 


16  100.0 

.775  Valid  cases 


Value  Frequency  Percent 


Mean 


4.563 


2 

3 

4 

5 

6 
7 

Total 
Std  dev 


1 

2 

6 

3 

2 

2 

16 

1.413 


6.3 

12.5 

37.5 
18.8 

12.5 
12.5 

100.0 


Valid  cases 


16 


16 


B-2-4 


BQ9  Supporting  information  and  actions  sought  from  crew 
Pretraining  Results: 


Rating 


Poor 

Marginal 

Acceptable 

Good 


Mean  3.250 

Post-training  Results: 

Rating 

Marginal 

Acceptable 

Good 

Very  Good 
Superior 


Mean  4.188 


Value  Frequency  Percent 


2 

3 

4 

5 

Total 
Std  dev 


3 

7 

5 

1 


18.8 

43.8 

31.3 

6.3 


16  100.0 

.856  Valid  cases 


Value  Frequency 


3  5 

4  7 

5  1 

6  2 

7  1 

Total  16 

Std  dev  1.: 


Percent 

31.3 

43.8 

6.3 

12.5 

6.3 

100.0 

Valid  cases 


16 


16 


BQIO  Crewmember  actions  mutually  cross  monitored 


Pretraining  Results: 

Rating 

Very  Poor 
Poor 

Marginal 

Acceptable 


Mean  2.688 


Value  Frequency 


1  2 

2  6 

3  3 

4  5 

Total  16 

Std  dev  1. 


Percent 

12.5 

37.5 
18.8 
31.3 

100.0 

Valid  cases 


Post-training  Results: 

Rating 


Poor 

Marginal 

Acceptable 

Good 

Very  Good 
Superior 


Value  Frequency  Percent 


2 

3 

4 

5 

6 
7 


1 

7 

4 

2 

1 

1 


6.3 

43.8 

25.0 

12.5 

6.3 

6.3 


Total 


16 


100,0 


Mean 


3 . 875  Std  dev 


1.310  Valid  cases 


16 


16 


B«2-5 


BQll  Supporting  information  and  actions  offered  by  crew 
Pretraining  Results: 


Rating 

Value 

Frequency 

Percent 

Very  Poor 

1 

1 

6.3 

Poor 

2 

4 

25.0 

Marginal 

3 

3 

18.8 

Acceptable 

4 

7 

43.8 

Good 

5 

1 

6.3 

Total 

16 

100.0 

Mean 

3.188 

Std  dev 

1.109 

Valid  cases 

Post- 

-training  Results: 

Rating 

Value 

Frequency 

Percent 

Marginal 

3 

5 

31.3 

Acceptable 

4 

7 

43.8 

Good 

5 

1 

6.3 

Very  Good 

6 

2 

12.5 

Superior 

7 

1 

6.3 

Total 

16 

100.0 

Mean 

4.188 

Std  dev 

1.223 

Valid  cases 

BQ12  Advocacy  and  assertion  practiced 
Pretraining  Results: 


Rating 

Value 

Frequency 

Percent 

Poor 

2 

3 

18.8 

Marginal 

3 

7 

43.8 

Acceptable 

4 

6 

37.5 

Total 

16 

100.0 

Mean 

3.188 

Std  dev 

.750 

Valid  cases 

Post- 

training  Results: 

Rating 

Value 

Frequency 

Percent 

Marginal 

3 

3 

18.8 

Acceptable 

4 

8 

50.0 

Good 

5 

5 

31.3 

Total 

16 

100.0 

Mean 

4.125 

Std  dev 

.719 

Valid  cases 

BQ13  Crew-level  after  action  reviews  accoirplished 


Pretraining  Results: 

Rating 

Value 

Frequency 

Percent 

Poor 

2 

2 

12.5 

Marginal 

3 

6 

37,5 

Acceptable 

4 

4 

25.0 

Good 

5 

3 

18.8 

Very  Good 

6 

Total 

1 

16 

6.3 

100.0 

Mean  3.688 

Std  dev 

1.138 

Valid  cases 

16 

Post-training  Results: 

Rating 

Value 

Frequency 

Percent 

Marginal 

3 

2 

12.5 

Acceptable 

4 

10 

62.5 

Good 

5 

3 

18.8 

Very  Good 

6 

Total 

1 

16 

6.3 

100.0 

Mean  4.188 

Std  dev 

.750 

Valid  cases 

16 
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Appendix  C 

Grade  Slip  and  ATM  Frequency  Tables 


C-1 


C-1-1 


MANEUVER/PROCEOURE  GRADE  SLIP  FOR  UH-60  ROM 


NO 

MANEUVER/PROCEDURE 

GR 

NO 

MANEUVER/PROCEDURE 

GR 

53 

EXTERNAL  LOAD  OPERATIONS 

79 

64 

INTERNAL  LOAD  OPERATIONS 

80 

55 

AERIAL  RADIO  RELAY 

81 

66 

ACTIONS  ON  CONTACT 

82 

<”> 

TERRAIN  FLIGHT  MISSION  PLANNING 

83 

TERRAIN  FLIGHT  NAVIGATION 

84 

TERRAIN  FLIGHT 

86 

60 

WIRE  OBSTACLES 

86 

61 

MASKING  AND  UNMASKING 

87 

62 

TERRAIN  FLIGHT  DECELERATION 

88 

63 

MAJOR  US/ALLIED  AND  THREAT 
EQUIPMENT  IDENTIFICATION 

mm 

89 

64 

TACTICAL  COMMUNICATION 
PROCEDURES  AND  ECCM 

90 

66 

TACTICAL  REPORT 

91 

66 

QUICK  FIX  MISSION 

92 

67 

FLAT  TURN/VCALIBRATED  FLIGHT 

93 

68 

ORAL  EVALUATION 

94 

69 

96 

70 

96 

71 

97 

72 

98 

73 

NOTES: 

74 

0  NVD  MANEUVER 
□  INSTRUMENT  MANEUVER 

0  STANDARDIZATION  MANEUVER 

ENTER  S  +  .  S.  S-.  OR  U  IN  GRADE  BLOCK.  IF 

GRADE  IS  S'  OR  U  DUE  TO  AIRCREW  COORDINATION 

76 

76 

77 

78 

1  s*  2,6 

Uh-  1  U  1  WU  basil;  UUAU  I  T  NUMBtMS. 

1 

AIRCREW  COORDINATION  SASIC  QUALITIES 


1. 

CREW 

CLI¬ 

MATE 

2. 

PLAN 

RE¬ 

HEARSE 

3 

DECI¬ 

SION 

TECH 

4. 

WORK 

LOAD 

5. 

UNEXP 

EVENTS 

6. 

INFO 

XFER 

7 

SIT 

AWARE 

8. 

COMM 

ACK 

9. 

INFO 

SOUGHT 

10, 

CROSS 

MON¬ 

ITOR 

n. 

INFO 

OF¬ 

FERED 

12. 

ADVOC/ 

ASSERT 

13. 

AAR 

1 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

1 

PAGE  2.  AIRCREW  COORDINATION  TRAINING  GRADE  SLIP 
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BATTLE-ROSTERED  CREW  EVALUATION/TRAINING  GRADE  SLIP 
For  use  of  this  form,  see  Aircraft  ATM;  the  proponent  agency  is  TRADOC 


BATTLE- 

ROSTERED 

CREW 

EXAMINEES/ 

TRAINEES 


EVALUATOR/ 

INSTRUCTOR 


CREW  DATA 


TOTAL  BATTLE-ROSTERED 
CREW  HOURS 


PURPOSE:  EVALUATION/TRAINING 


TIME  TODAY: 


TYPE  AIRCRAFT: _ 

CREW  TASK  1 _ D/N/NVD 

CREW  TASK  2 _ D/N/NVD 

CREW  TASK  3 _ D/N/NVD 

CREW  TASK  4 _ D/N/NVD 

CREW  TASK  5 _ D/N/NVD 

DAY  NIGHT  WX 


DATE  DESIGNATED  A  BATTLE- 
ROSTERED  CREW: 


CUMULATIVE  TIME: 


CREW  TASK  6  _ D/N/NVD 

CREW  TASK  7  _ D/N/NVD 

CREW  TASK  8  _ D/N/NVD 

CREW  TASK  9  _ D/N/NVD 

CREW  TASK  10 _ D/N/NVD 

SIMULATOR  NVG  NVS 


EVALUATOR/INSTRUCTOR  RECOMMENDATIONS 


O  (ISSUE)  (VALIDATE)  CREW  QUALIFICATIONS 


a  (SUSPEND)  (REVOKE)  CREW  QUALIFICATIONS 


a  REQUIRES  ADDITIONAL  (FLIGHT)  (ACADEMIC)  (SIMULATION  DEVICE)  TRAINING 


□  SEE  BACK  FOR  COMMENTS 


I  HAVE  DEBRIEFED  THE  EXAMINEES/TRAINEES  AND  INFORMED  THEM  OF  THEIR  STATUS. 

EVALUATOR‘S/INSTRUCTOR*S  SIGNATURE: _ 


WE  HAVE  BEEN  DEBRIEFED  BY  THE  EVALUATOR/INSTRUCTOR  AND  UNDERSTAND  OUR  CURRENT 
STATUS. 

PC’S  SIGNATURE: _ 

PI’S  SIGNATURE: _ 

NONRATED  CREW  MEMBER’S  SIGNATURES:  _ 


OVERALL  GRADE  FOR  THIS  FLIGHT  IS:  S  U  NA 


COMMENTS 


APPENDIX  C-2 


ATM  FREQUENCY  TABLES 


TIOOO  Crew  Mission  Briefing 

Pretraining  Results: 


Grade 

Value 

Frequency 

Percent 

U 

0 

5 

31.3 

S- 

1 

7 

43.8 

S 

2 

4 

25.0 

Total 

16 

100.0 

Mean  .938 

Std  dev 

.772 

Valid  cases 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

S- 

1 

2 

12.5 

S 

2 

10 

62.5 

S+ 

3 

4 

25.0 

Total 

16 

100.0 

Mean  2.125 

Std  dev 

.619 

Valid  cases 

T1004  DA  Form  5701“R(PPC) 

Pre training  Results : 

Grade 
U 

s-- 

s 

s+ 

Total 

Mean  1.750  Std  dev 

Post- training  Results : 

Grade 

S- 
S 

S+ 

Total 

Mean  2.063  Std  dev 


Frequency  Percent 


1 

6.3 

3 

18.8 

11 

68.8 

1 

6.3 

16 

100.0 

.683  Valid  cases  16 


Frequency  Percent 


1 

6.3 

13 

81.3 

2 

12.5 

16 

100.0 

.443  Valid  cases  16 


Value 

0 

1 

2 

3 


Value 

1 

2 

3 


C-2-1 


T1007  Engine  start-up,  run-up,  T/O,  Land  Checks 

Pretraining  Results: 


Grade 

Value 

Frequency 

Percent 

U 

0 

2 

12.5 

S- 

1 

1 

6.3 

S 

2 

13 

81.3 

- 

Total 

16 

100.0 

Mean 

1.688 

Std  dev 

.704 

Valid  cases 

Post- training 

Results : 

Grade 

Value 

Frequency 

Percent 

S- 

1 

3 

18.8 

S 

2 

11 

68.8 

S+ 

3 

2 

12.5 

Total 

16 

100.0 

Mean 

1.938 

Std  dev 

.574 

Valid  cases 

T1016  Hover  Power  Check 


Pretraining  Results: 

Grade 

U 

S- 

Mean  .313 


Value  Frequency 


0  11 

1  5 

Total  16 

Std  dev  .479 


Percent 

68.8 

31.3 

100.0 

Valid  cases  16 


Post-training  Results: 

Grade 

U 

s- 

s 

s+ 


Mean  1,688 


Value  Frequency 


0  2 

1  4 

2  7 

3  3 

Total  16 

Std  dev  .946 


Percent 

12.5 

25.0 

43.8 

18.8 

100.0 

Valid  cases  16 


C-2-2 


T1018 


Perform  VMC  Takeoff 


Pre training  Results: 


Grade 

Value 

Frequency 

Percent 

S 

2 

16 

100.0 

Total 

16 

100.0 

Mean  2.000 

Std  dev 

.000 

Valid  cases 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

S 

2 

13 

81.3 

S+ 

3 

3 

18.8 

Total 

16 

100.0 

Mean  2,188 

Std  dev 

.403 

Valid  cases 

T1023  Fuel  Management  Procedures 

Pretraining  Results: 


Grade 

U 

S- 

S 


Mean  .250  Std 

Post-training  Results: 

Grade 

U 

S- 

S 


Mean  1.313  Std 


Value 

Frequency 

Percent 

0 

13 

81.3 

1 

2 

12.5 

2 

1 

6.3 

Total 

16 

100.0 

iV 

.577 

Valid  cases 

Value 

Frequency 

Percent 

0 

4 

25.0 

1 

3 

18.8 

2 

9 

56.3 

Total 

16 

100.0 

:V 

.873 

Valid  cases 

16 


16 


C-2-3 


T1028 


VMC  Approach 


Pretraining  Results : 

Grade 

Value 

Frequency 

Percent 

S- 

1 

2 

12.5 

S 

2 

14 

87.5 

Total 

16 

100.0 

Mean  1.875 

Std  dev 

.342 

Valid  cases 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

S- 

1 

2 

12.5 

S 

2 

12 

75.0 

S+ 

3 

2 

12.5 

Total 

16 

100.0 

Mean  2.000 

Std  dev 

.516 

Valid  cases 

T1068  Emergency  Procedures 


Pre training  Results: 


Grade 

U 

S- 

S 

S+ 


Mean  1,188 


Value  Frequency 


0  5 

1  4 

2  6 

3  1 

Total  16 

Std  dev  .981 


Percent 

31.3 

25.0 

37.5 

6.3 

100.0 

Valid  cases  16 


Post-training  Results: 

Grade 

u 

s- 

s 

s+ 


Mean  1.688 


Value  Frequency 


0  2 

1  4 

2  7 

3  3 

Total  16 

Std  dev  .946 


Percent 

12.5 

25.0 

43.8 

18.8 

100.0 

Valid  cases  16 


C“2-4 


T1076 


Perform  Radio  Navigation 


Pretraining  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

1 

6.3 

S- 

1 

2 

12.5 

S 

2 

12 

75.0 

S+ 

3 

1 

6.3 

Total 

16 

100.0 

Mean  1.813 

Std  dev 

.655 

Valid  cases 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

S 

2 

14 

87.5 

S+ 

3 

2 

12.5 

Total 

16 

100.0 

Mean  2.125 

Std  dev 

.342 

Valid  cases 

T1081  Perform  Nonprecision  Approach 


Pretraining  Results: 

Grade 

U 

S- 

S 

S+ 


Mean  1,188 

Post-training  Results: 

Grade 

U 

S- 

S 

S+ 


Mean  1.688 


Value  Frequency 


0  6 

1  3 

2  5 

3  2 

Total  16 

Std  dev  1.109 


Value  Frequency 


0  1 

1  6 

2  6 

3  3 

Total  16 

Std  dev  .873 


Percent 

37.5 
18.8 
31.3 

12.5 

100.0 


Valid  cases 


Percent 

6.3 

37.5 

37.5 

18.8 

100.0 

Valid  cases 


16 


16 


C-2-5 


T1083  Perform  IMC  : 

Procedures  (VHIRP) 

Pretraining  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

2 

12.5 

S- 

1 

3 

18.8 

S 

2 

7 

43.8 

S+ 

3 

4 

25.0 

Total 

16 

100.0 

Mean  1.813 

Std  dev 

,981 

Valid 

cases 

16 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

1 

6.3 

s- 

1 

2 

12.5 

s 

2 

8 

50.0 

s+ 

3 

5 

31.3 

Total 

16 

100.0 

Mean  2.063 

Std  dev 

.854 

Valid 

cases 

16 

T1095  Operate  Aircraft  Surviv2U3ilitY  Equipment 

Pre training  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

2 

12.5 

S- 

1 

3 

18.8 

S 

2 

11 

68.8 

Total 

16 

100.0 

Mean  1.563 

Std  dev 

.727 

Valid 

cases 

16 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

•  S- 

1 

1 

6.3 

S 

2 

14 

87.5 

S+ 

3 

1 

6.3 

Total 

16 

100.0 

Mean  2.000 

Std  dev 

.365 

Valid 

cases 

16 

c- 

2-6 

T2  0  0  8  Evas i ve  Maneuvers 

Pretraining  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

2 

12.5 

S- 

1 

4 

25.0 

S 

2 

10 

62.5 

Total 

16 

100.0 

Mean  1.500 

Std  dev 

.730 

Valid  cases 

16 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

1 

6.3 

S- 

1 

4 

25.0 

S 

2 

8 

50.0 

S+ 

3 

3 

18.8 

Total 

16 

100.0 

Mean  1.813 

Std  dev 

.834 

Valid  cases 

16 

T2009  Perform  Multi 

A/C  Operations 

Pretraining  Results: 

Grade 

Value 

Frequency 

Percent 

S- 

1 

7 

43.8 

S 

2 

9 

56.3 

Total 

16 

100.0 

Mean  1.563 

Std  dev 

.512 

Valid  cases 

16 

Post- training  Results : 

Grade 

Value 

Frequency 

Percent 

S- 

1 

4 

25.0 

S 

2 

10 

62.5 

S+ 

3 

2 

12.5 

Total 

16 

100.0 

Mean  1.875 

Std  dev 

.619 

Valid  cases 

16 

2-1 

T2016  Perform  External  Load  Operations 

Pre training  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

2 

12.5 

S- 

1 

5 

31.3 

S 

2 

5 

31.3 

S+ 

3 

1 

6.3 

• 

3 

18.8 

Total 

16 

100.0 

Mean  1,385 

Std  dev 

.870 

Valid 

cases 

13 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

S- 

1 

2 

12.5 

S 

2 

12 

75.0 

S+ 

3 

2 

12.5 

Total 

16 

100,0 

Mean  2.000 

Std  dev 

.516 

Valid 

cases 

16 

T2078  Terrain  Flight 

Mission  Planning 

Pretraining  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

1 

6.3 

s- 

1 

10 

62.5 

s 

2 

5 

31.3 

Total 

16 

100.0 

Mean  1.250 

Std  dev 

.577 

Valid 

cases 

16 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

S- 

1 

1 

6.3 

S 

2 

10 

62.5 

S+ 

3 

5 

31.3 

Total 

16 

100.0 

Mean  2.250 

Std  dev 

.577 

Valid 

cases 

16 

c- 

2-8 

T2079  Terrain  Flight  Navigation 

Pretraining  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

4 

25.0 

S- 

1 

9 

56.3 

s 

2 

3 

18.8 

Total 

16 

100.0 

Mean  .938 

Std  dev 

,680 

Valid 

cases 

16 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

1 

6.3 

S- 

1 

3 

18.8 

s 

2 

7 

43.8 

S4- 

3 

5 

31.3 

Total 

16 

100.0 

Mean  2.000 

Std  dev 

.894 

Valid 

cases 

16 

T2081  Perform  Terrain  Flight 

Pretraining  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

1 

6.3 

S- 

1 

1 

6.3 

s 

2 

14 

87.5 

Total 

16 

100.0 

Mean  1.813 

Std  dev 

.544 

Valid 

cases 

16 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

S- 

1 

1 

6.3 

S 

2 

14 

87.5 

s+ 

3 

1 

6.3 

Total 

16 

100.0 

Mean  2.000 

Std  dev 

.365 

Valid 

cases 

16 

c- 

2-9 

OVGRD 


Overall  Grade 


Pre training  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

6 

37.5 

S- 

1 

8 

50.0 

S 

2 

2 

12.5 

Total 

16 

100.0 

Mean  .750 

Std  dev 

.683 

Valid  cases 

Post-training  Results: 

Grade 

Value 

Frequency 

Percent 

U 

0 

2 

12.5 

S- 

1 

2 

12.5 

S 

2 

9 

56.3 

S+ 

3 

3 

18.8 

Total 

16 

100.0 

Mean  1.813 

Std  dev 

.911 

Valid  cases 
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Appendix  D 

Aircrew  Coordination  Training  Validation  Testbed 

Exit  Interviews 


CREWMEMBER  EXIT  INTERVIEW 


I.  Course  Administration 

1.  Was  the  classroom  appropriately  arranged  for  the  nuxnber  of 
students  present? 

Summary:  Yes.  General  agreement  that  the  classroom 

arrangement  was  comfortable  and  effective. 

Specific  Comments : 

Liked  the  horseshoe  formation.  It  gave  everyone  a  front 
row  seat. 

Keep  the  horseshoe  arrangement,  in  the  classroom. 

Improve  horseshoe  arrangement  to  allow  more  direct  route 
to  exits. 

The  horseshoe  arrangement  is  a  good  way  to  facilitate 
discussions . 

Instructor  on  end  of  horseshoe  made  class  more  formal. 
Prefer  that  the  screen  and  instructor  be  more  centered. 
The  television  screen  wasn't  placed  very  well. 

Conclusion:  Recommended  that  classrooms  be  arranged  in  a 

horseshoe  or  other  formation  that  promotes  intervisibility, 
discussion,  and  a  seminar  environment. 


2.  Was  the  nuniber  of  students  in  the  class  about  the  right  size 
for  this  training? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

Five  to  eight  crews  was  about  right.  No  more  than  eight 
crews . 

Could  have  fewer  but  not  more  crews  in  a  class. 

Eight  crew  class  size  helped  have  better  discussions. 
More  crews  would  take  more  time  to  teach. 

Conclusion:  Recommended  that  the  student  course  class  size 
be  limited  to  16  rated  and/or  nonrated  crewmembers. 
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3.  Did  the  videotaping  of  the  classroom  instruction  detract  from 
or  enhance  the  classroom  environment?  If  so,  how? 

Summary:  Mixed  opinion. 

Specific  Comments: 

At  first,  people  were  reluctant  to  discuss  personal 
experiences  and  actual  incidents  but  eventually  they 
opened  up. 

Would  get  more  discussions  without  the  camera. 

It  enhanced  research  but  inhibited  some  comments  and  may 
have  inhibited  some  incident  sharing. 

At  first,  it  was  intimidating  but  eventually,  it  didn't 
make  a  difference. 

Enhanced  things  for  the  [research]  test  team  but  it 
inhibited  us . 

Helped  clean  up  language. 

It  probably  decreased  the  "informal"  language  a  bit. 
Conclusions: 

Videotaping  classroom  instruction  tends  to  initially 
distract  and  generally  inhibit  open  discussion. 
Recommended  that  classroom  instruction  not  be  videotaped 
except  for  instructor  methods  of  instruction  (MOI) 
training  as  required. 


4 .  Was  the  instructional  staff  properly  prepared  to  conduct  the 
course?  If  not  prepared,  what  deficiencies  did  you  note? 

Summary:  Yes.  General  agreement  that  instructors  were  well 
prepared  given  the  limited  time  available. 

Specific  Comments : 

Strengths : 

IPs  are  better,  more  accustomed  to  teaching. 

Good  to  have  UTs  and  IPs  involved  in  team  teaching  the 
course,  that  way  UTs  receive  a  training  benefit  too. 

IPs  seemed  to  be  more  accustomed  to  instructing  compared 
to  UTs.  IPs  gave  examples,  UTs  mostly  read  from  the 
book. 

All  instructors  followed  the  course  outline  fairly 
closely. 

UTs  and  IPs  did  a  good  job  given  the  short  preparation 
time — they  were  "believers"  in  the  system. 

Cross  training  of  crews  between  units  was  good  as  a 
reinforcing  technique. 

CW3  Lear  is  a  natural  instructor;  was  always  prepared; 
exhibited  a  relaxed  style;  balanced  humor  and 
seriousness . 

It's  good  to  use  our  own  instructors  in  the  training. 
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Weaknesses : 

Tendency  to  read  from  the  instructor's  outline  and 
slides.  Some  instructors  didn't  seem  to  have  enough 
background  and  experience. 

Some  questions,  for  example,  CRL  progression  and  battle 
rostering,  had  to  be  answered  by  the  Fort  Rucker  trainer 
team  [CW4  Sheehan  and  Mr.  Pawlik] . 

I  thought  that  their  training  should  have  been  more  in- 
depth.  They  needed  more  time  to  prepare. 

Conclusions : 

Recommended  that  unit  instructor  pilots  implement  crew 
coordination  training. 

IPs  must  prepare  crew  coordination  qualified  unit 
trainers  to  assist  them  in  team  teaching  unit  crews.  IPs 
are  the  principal  instructors  within  IP-UT  instructor 
teams  and  instill  a  positive  attitude  toward  crew 
coordination. 

Instructors  need  at  least  2  hours  of  preparation  time  for 
every  1  hour  of  classroom  instruction. 


5 .  Were  the  facilities  adequate  during  the  simulator  phase  of 
training?  If  not,  what  was  unsatisfactory? 

Summary:  Mixed  opinion. 

Specific  Comments : 

Some  of  the  briefing  rooms  were  too  small  to  spread  out 
maps  and  flight  planning  materials. 

Facilities  were  excellent  compared  to  what  is  provided  in 
the  field  for  mission  planning. 

It  was  good  to  have  separate  rooms  for  premission 
planning  and  AAR. 

Two  crews  planning  together  for  training  missions  was  not 
a  problem. 

Should  reduce  the  size  of  the  maps  and  laminate  them  so 
can  use  permanent  markers.  Water-based  markers  rub  off 
from  map  case  covers. 

Have  coffee  available  during  planning  and  after-  action 
review. 

One  military  and  one  civilian  10  didn't  role  play  the 
crew  chief  position  very  well.  Many  SOP  items  and 
expected  crew  chief  actions  were  not  provided. 

It  would  be  better  to  use  unit  lOs  who  are  more  familiar 
with  unit  SOPs  and  expected  duties.  Unit  lOs  normally  go 
to  the  simulator  with  the  crews. 

Simulator  computer  crashes  were  distracting  and 
frustrating. 
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Conclusions : 

Recoramended  that  premission  planning  and  after-action 
review  activities  be  kept  separate. 

Use  the  flight  planning  materials  and  administrative 
resources  normally  available  in  the  unit  and  installation 
flight  simulator  facility. 

Emphasize  the  need  for  military  and  civilian  flight 
simulator  lOs  to  role  play  (for  example,  other 
crewmembers,  other  aircraft,  and  tactical  units)  during 
simulator  missions.  Provide  10  scripts  for  scenario 
events  and  required  lO-crew  interaction. 


6.  What  changes  do  you  recommend  to  improve  the  administration 
of  future  courses? 

Summary:  General  agreement  on  the  types  of  changes  needed  to 

improve  the  course. 

Specific  Comments : 

Schedule : 

See  related  responses  at  item  II,  16. 

Consider  increasing  the  number  of  hours  from  18  to  20  and 
scheduling  the  course  over  five  half  days  instead  of 
three  full  days.  This  would  spread  out  the  instruction 
and  allow  more  time  for  understanding. 

Offer  full-day  and  half-day  scheduling  options.  Some 
units  want  crews  to  perform  other  duties  during  training. 
Provide  more  advanced  notice  and  information  about  the 
course  (for  example,  the  requirement  for  tactical 
missions  in  the  flight  simulator) . 

Explore  opportunities  to  shorten  the  course.  Some  topics 
seem  to  be  common  sense. 

Conduct  the  course  off-post  if  using  the  full-day 
schedule . 

Obtain  command  emphasis  to  ensure  that  crews  are 
available  for  training  without  competing  demands  on  their 
time . 

Block  the  unit  training  schedule  well  in  advance. 

Crew  Composition: 

Consider  mixing  and  matching  crews  throughout  the  course 
to  promote  crew  coordination  standardization  rather  than 
familiarization. 

Mix  crews  for  the  training  missions.  Match  the  crews  per 
the  unit's  battle  roster  for  the  evaluation  missions. 
Explore  the  difference  in  performance  between  non-battle 
rostered  crews  with  crew  coordination  training  and  battle 
rostered  crews  without  crew  coordination  training. 
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Types  of  Missions: 

Integrate  an  actual  aircraft  flight  into  the  training-- 
maybe  not  the  final  ride.  Artificial  nature  of  the 
flight  simulator  can  be  distracting  and  introduce  a 
simulator  mindset  (e.g.,  should  I  really  set  the  exterior 
or  anticollision  lights  or  not?) . 

Consider  adding  a  night  flight  in  the  actual  aircraft  or 
simulator — with  battle-rostered  crews  only.  Prefer 
aircraft  over  simulator  for  NVD  missions. 

Academics : 

Consider  adding  a  review  of  MOI  techniques  for  the  UTs. 
IPs  could  benefit  from  Instructor  Training  Course 
experience  to  improve  their  classroom  technique. 

Need  to  introduce  crew  coordination  in  a  top-down  fashion 
within  the  units.  Start  with  the  commander. 

Would  be  good  to  have  a  UT  dedicated  to  teaching  crew 
coordination. 

There  needs  to  be  a  program  to  "qualify"  instructors  to 
teach  this  course. 

Add  more  "key  points"  in  the  student  outline.  Spent  too 
much  time  filling  in  the  blanks  while  following  the 
lesson  plan. 

Conclusions : 

Included  full-day  and  half-day  classroom  training 
schedule  options  in  the  course. 

Recommended  that  units  use  the  full-day  training  option 
with  students  placed  on  excused  duty  status.  If  excused 
duty  status  is  not  possible,  then  recommended  the  half¬ 
day  training  option  for  five  days. 

Recommended  that  crew  coordination  training  be  placed  on 
the  training  schedule  with  the  same  command  emphasis  as 
field  training  exercises. 

Included  an  option  to  mix  crewmembers  rather  than  rigid 
battle-roster  for  training  missions.  Emphasized  that  crew 
coordination  training  is  not  learning  how  to  operate  with 
a  particular  crewmember;  it's  a  set  of  standardized  oper¬ 
ating  procedures/techniques  to  be  used  in  any  situation. 
Expanded  the  course  to  include  a  pre-training  mission  and 
a  course  completion  evaluation  mission  in  the  simulator 
or  aircraft. 

Modified  the  course  to  allow  for  local  consideration  of  a 
fourth  training  mission  with  options  for  NVD,  aircraft 
instead  of  simulator,  and/or  mixing  crewmembers. 

Provided  student  read-ahead  materials. 

Included  a  review  of  MOI  principles  in  the  Trainer 
Course . 

Expanded  the  Student  Handout  to  include  abbreviated 
talking  points. 

Recommended  that  if  units  decide  to  designate  a  unit 
trainer  for  crew  coordination,  that  such  appointments 
clearly  specify  that  crew  coordination  is  an  IP  program. 
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II.  Course  Structure 

1.  Was  the  course  well  organized  in  terms  of  subject  flow? 

Summary:  Mixed  opinion.  See  comments  and  conclusions  for 
items  a  through  j . 

a .  overview? 

Specific  Comment:  It  seemed  to  me  that  we  spent  too  much 
time  on  the  overview. 

b.  history? 

Specific  Comments: 

Shorten  the  amount  of  time  spent  on  this. 

It's  interesting  but  don't  spend  too  much  time  on  it. 

c.  structure  of  the  course? 

Specific  Comments : 

The  whole  course  seemed  like  everything  was  a  shot  gun 
effect.  Some  things  seemed  scattered  and  not  related,  and 
other  topics  seemed  redundant. 

It  was  hard  to  follow  material  without  insight  into  how 
the  topics  were  related. 

Course  structure  information  is  of  little  value  and 
should  be  covered  quickly.  Need  to  get  to  the  heart  of 
the  course  quickly. 

Good  use  of  a  video  segment  to  drive  home  an  image  of  why 
I  need  to  be  in  this  class.  That  got  my  attention. 

Need  to  set  the  course  up  to  help  crews  who  are  flying 
together  for  the  first  time. 

Conclusion:  Modified  the  course  to  shorten  the  introduction 
and  introduce  the  crosswalk  chart  of  related  crew 
coordination  topics  as  a  means  to  get  to  the  heart  of  the 
instruction  early. 

d.  crew  coordination  model? 

Specific  Comments : 

Spent  too  much  time  on  models  and  the  logo.  Wanted  to 
get  going  faster. 

The  cover  logo  is  not  the  same  as  the  crew  coordination 
model — found  this  to  be  confusing. 

I  thought  the  logo  and  the  model  were  good.  Our 
instructors  didn't  overuse  them. 

Good.  Easy  to  understand.  Clear  and  understandable. 

The  term  "models"  was  used  inconsistently  throughout  the 
course . 
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e.  crew  coordination  elements? 

No  problems  or  comments. 

f.  Basic  Qualities? 

Specific  Comments: 

Hard  to  remember  13  Basic  Qualities.  Alright  as  long  as 
not  required  to  memorize. 

Made  you  think  of  things  that  you  normally  take  for 
granted  when  you  fly  with  a  new  crewmember. 

The  BQs  are  important  by  themselves,  but  the  models  and 
so  forth  didn't  help  me. 

g.  crew  coordination  objectives? 

No  problems  or  comments. 

h.  definition,  discussion,  effectiveness  factors,  and 
exan^les  of  Basic  Qualities? 

Summary:  Mixed  opinion. 

Specific  Comments : 

Emphasizing  the  acceptable  level  as  the  initial  training 
objective  made  sense. 

It  was  good  that  the  scales  provided  for  an  "acceptable" 
performance  level,  rather  than  just  forcing  everyone  to 
excel. 

Showed  a  lot  of  difference  between  superior  and  very  poor 
ratings . 

Once  fielded,  should  strive  for  superior  instead  of 
acceptable  performance  levels. 

Think  should  emphasize  superior.  Disagree.  Acceptable 
is  simple,  direct,  and  doable/  superior  may  take  too  much 
initial  training  time. 

Train  to  acceptable  with  sights  on  superior. 

Work  toward  superior  in  unit  continuation  training. 
Emphasize  the  superior  more.  It's  important  to  have  high 
standards . 

Highlight  the  superior  factors  in  the  classroom. 

Emphasize  the  acceptable  performance  level. 

Conclusion:  Established  the  acceptable  level  of  performance 

as  the  training  goal  for  initial  crew  coordination  training. 

i .  phase  review? 

Specific  Comments : 

See  related  responses  at  item  III,1. 

If  there  were  a  written  test  at  the  end  of  the  course, 
then  we  would  have  paid  more  attention  in  class. 

Consider  adding  a  written  test  on  the  academics. 
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Conclusion:  Added  sample  oral  examination  questions  for 
evaluator  debriefings  that  could  be  used  as  test  questions 
for  the  academic  phase. 

j .  hands-on  simulator  application? 

Specific  Comments: 

See  related  responses  at  item  III, 2. 

A  brief  introduction  is  needed  prior  to  the  first 
simulator  evaluation  flight — had  no  idea  what  to  expect 
that  first  ride. 

Uncertainty  over  the  first  evaluation  ride  created  a 
negative  attitude  toward  the  course. 

First  evaluation  ride  had  a  positive  effect  on  learning 
during  the  classroom.  You  can  draw  on  the  experience  in 
class. 

After  the  first  ride  I  could  really  see  the  difference. 
Its  a  psychological  advantage  to  fail  so  badly,  then 
improve  so  much. 

Need  both  classroom  and  simulator.  Classroom  itself  is 
not  enough. 

Simulator  allows  for  various  problems  and  stressful 
situations  to  be  thrown  at  you. 

Simulator  training,  despite  its  lack  of  realism,  helps  in 
training  for  actual  aircraft. 

Simulator  (lacking  GPS)  forces  you  back  to  basic 
navigation  techniques. 

The  simulator  is  valuable,  but  we  need  help  to  make  sure 
we  do  it  in  the  aircraft.  Use  no-notice  UT  rides  to 
reinforce  crew  coordination  in  the  aircraft. 

Conclusions : 

Both  classroom  instruction  and  simulator  or  aircraft 
hands-on  application  are  required  in  the  course. 
Incorporated  the  pre-training  mission  into  the  course 
because  it  helps  students  relate  to  the  subject  matter 
and  to  focus  on  important  points  in  the  classroom 
instruction.  Incorporated  read-aheads  to  orient  students 
to  the  course. 


2.  Were  the  subjects  presented  appliccible  to  your  job  as  an  Army- 
aviator? 

Summary:  Yes.  General  agreement  with  one  exception. 

Specific  Comment:  Flight  standards  should  provide  units  with 
the  correct  priorities  for  mission  planning  tasks,  rather 
than  having  the  crews  develop  them. 

Conclusion:  No  action  recommended  due  to  the  range  of  METT-T 
considerations  across  units. 
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3.  Were  the  subjects  well  developed  so  that  you  are  confident 
that  you  understand  the  material? 

Summary:  Yes.  General  agreement. 

Specific  Comments : 

May  have  developed  some  subjects  too  much.  Some  topics 
seem  repetitive. 

Maybe  too  much  time  was  spent  on  some  examples. 

Reduce  .the  amount  of  lecture  and  substitute  class 
analysis  of  case  studies. 

Develop  more  practical  exercises  for  student  involvement. 

Conclusion:  Expanded  the  current  practical  exercises  to 
include  tactical  examples. 


4.  Was  the  interrelationship  between  Crew  Coordination  Elements 
in  the  ATM  Tasks,  the  Basic  Qualities,  and  the  Crew 
Coordination  Objectives  clearly  established? 

Summary:  Mixed  opinion.  Majority  agreed  that  the  crosswalk 
chart  would  be  a  benefit. 

Specific  Comments : 

Still  had  difficulty  at  the  end  of  the  course  relating 
objectives,  qualities,  etc. 

Need  to  provide  students  with  a  crosswalk  chart  showing 
relationships  among  objectives,  qualities,  and  elements 
[given  only  to  IPs] — provide  it  two-thirds  of  the  way 
through  the  course  or  at  the  end. 

The  crosswalk  chart  ties  it  all  together  [students  had  to 
do  this  on  their  own  or  remain  confused] — blow  it  up  as  a 
wall  chart  to  be  displayed  throughout  the  course. 

Hard  to  follow  various  numbers  of  objectives,  qualities, 
etc . 

It's  a  bit  too  many  numbers — 6  elements,  13  BQs,  5  CCOs . 
Reached  a  saturation  point.  Stress  only  what  is 
important  in  the  cockpit  and  forget  the  rest  (e.g., 
objectives) . 

Crosswalk  chart  would  be  good  to  have  as  a  handout  or 
hang  it  up  in  class  so  we  could  use  it  all  the  time. 
Suggest  using  the  crosswalk  chart  instead  of  the  "target" 
model  as  a  training  aid. 

The  relationships  were  very  clearly  established,  but  at 
the  user  level,  the  relationships  are  not  relevant  and 
would  not  affect  the  way  you  fly. 

Basic  Qualities  are  important,  but  not  the  other 
relationships  to  elements,  ATM  tasks,  etc. 

Conclusion:  Need  to  provide  the  crosswalk  chart  during  class 
to  keep  students  on  track  with  the  instruction.  Included  the 
crosswalk  chart  in  the  Student  Handout  materials. 
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5 .  Were  the  Basic  Qualities  V7ell  defined  and  explained? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

The  Basic  Qualities  made  you  think  about  things  you  might 
otherwise  take  for  granted. 

Could  combine  some  Basic  Qualities  that  appear  to  overlap 
one  another  or  relate  to  the  same  end. 

Thirteen  is  a  good  start  point  for  branching  out;  there 
is  a  tendency  for  BQs  to  intersect. 

Some  of  them  overlap  but  it  depends  on  the  situation;  I 
can  see  how  they  are  interdependent. 

They  were  good.  There  was  the  right  amount  of  them. 

I'm  not  sure  why  the  BQ  of  cross-monitor  is  the  same  as 
the  CCO  of  cross-monitor.  They  seem  about  the  same. 

Some  seem  to  fuse  together  (for  example,  information 
sought  and  information  offered) . 

They  seemed  fine.  Just  about  the  right  number. 

Well  thought  out.  Some  overlap,  but  just  the  right 
amount . 

Conclusion:  Emphasized  the  importance  of  Basic  Qualities  to 
teaching  and  evaluating  crew  coordination. 


6.  Were  the  effectiveness  [rating]  factors  clearly  linked  to  the 
Basic  Qualities? 

Summary:  Mixed  opinion. 

Specific  Comments : 

Yes,  they  seemed  to  be. 

Effectiveness  factors  were  not  covered  thoroughly  in 
class . 

I  thought  that  some  of  the  superior  descriptions  were  not 
doable.  They  require  too  much  training  time. 

We  needed  more  time  to  understand  the  effectiveness 
factors.  We  just  read  them  in  class  and  went  on.  More 
discussion  is  necessary. 

Conclusions : 

The  terms  effectiveness  factors  and  rating  factors  were 
used  interchangeably  during  course  development  and 
testing. 

Modified  course  materials  to  consistently  refer  to  these 
factors  as  "rating  factors"  and  emphasized  their 
relationship  to  the  Basic  Qualities  and  behavioral 
anchored  ratings  that  they  support. 
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7.  Were  the  Crew  Coordination  Objectives  well  demonstrated  in 
terms  of  case  studies? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

Need  some  UH-60  Broken  Wing  award  examples. 

They  were  pretty  realistic  and  made  you  think  about 
similar  situations. 

Case  studies  were  good.  They  got  the  class  more  involved 
in  discussing  situations  similar  to  the  cases.  Should 
call  on  more  people  to  participate  in  discussions. 

Should  have  more  discussion  time  for  case  studies. 

Could  use  a  greater  number  and  more  varied  case  studies. 
Emphasize  emergency  procedures  more  to  illustrate  the 
role  of  each  crewmember.  We  need  to  learn  how  to  use  our 
crewmembers  better  in  emergency  situations. 

Need  a  case  for  decision  techniques. 

Develop  case  studies  with  different  endings  due  to  the 
effects  of  crew  coordination. 

Conclusions : 

Recommended  that  the  course  proponent  and  instructors 
continuously  improve  the  course  by  adding  both  positive 
and  negative  case  study  examples. 

Identified  candidate  segments  from  the  testbed  missions 
that  support  development  of  "branching"  scenarios  or  that 
could  be  used  to  enrich  the  video  case  studies. 

8.  Were  the  Crew  Coordination  Objectives  well  demonstrated  in 
terms  of  video  segments? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

'They  were  good.  Crew  coordination  could  apply  to  almost 
any  accident. 

They  seemed  to  really  emphasize  the  importance  of  crew 
coordination  (e.g.,  fast  rope  example). 

Could  benefit  from  videos  that  could  be  stopped  for 
questions  and  have  branching  to  different  endings  (video 
comparisons  of  good  and  bad  actions) . 

Need  more  video  segments. 

Video  taping  situations  recreated  in  the  simulator  is  a 
good  approach. 

Question  whether  the  CH-54  miscalculation  of  gross  weight 
was  a  crew  coordination  error:  It  was  more  an  individual 
error.  [Discussed  need  for  cross  monitoring.] 

I  think  the  videos  should  be  actual  or  reenactments  in 
the  aircraft.  Reconstructions  in  the  simulator  are  second 
best . 

Need  more  crew  chief  oriented  videos.  The  fast  rope 
video  would  be  good  for  crew  chiefs. 
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Conclusions : 

The  Army  should  use  in-house  resources  to  develop  video 
segments  by  recreating  cases  in  the  simulator  or 
aircraft . 

Recommended  specific  segments  from  the  testbed  videotapes 
that  have  crewmember's  consent  for  addition  to  the 
course . 

Stress  the  crew  error  aspect  of  the  CH-54  accident  video 
segment . 


9.  Was  there  about  the  right  mix  of  video  and  written  case 

studies  to  help  you  understand  the  Basic  Qualities  and  Crew 
Coordination  Objectives? 

Summary:  Need  more  videos. 

Specific  Comments: 

Video  examples  really  assist  in  illustrating  the  various 
Basic  Qualities — the  more  video  segments  the  better. 

The  video  and  audio  tapes  really  helped.  It  didn't  matter 
that  the  examples  were  from  different  aircraft. 

It  would  be  good  to  read  a  case  study,  then  see  it  on  the 
video.  That  would  be  helpful  to  better  understand 
everything  that  is  going  on. 

It's  good  to  read  and  then  see;  then  you  get  a  real  feel 
for  it. 

Conclusion:  Recommend  that  additional  videos  be  developed  to 
visually  illustrate  the  case  studies. 


10.  Was  the  Student  Handout  satisfactory? 

Summary:  Mixed  opinion.  See  comments  and  conclusions  for 
items  a  through  c. 

a.  Did  it  assist  you  in  following  the  instructor's 
presentation? 

Specific  Comments: 

Yes,  but  make  it  a  separate  book  that  can  be  written  in-- 
didn't  like  having  to  flip  back  and  forth  in  the  same 
book. 

Sometimes  I  got  confused  about  which  section  was  which. 
Would  have  preferred  to  have  the  handout  separate  from 
the  book.  That  way  it  would  be  easier  to  get  to  the 
various  sections  in  the  book  and  still  keep  my  place  in 
the  course  presentation. 

Found  it  hard  to  follow — got  lost  as  to  where  the 
instructor  was  at  in  a  particular  discussion  versus  the 
reference  in  the  handout. 
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Not  always;  unable  to  track  with  instructors  comments. 

Too  much  information  to  listen  and  write. 

Conclusions : 

Modified  the  Instructor's  Guide  to  indicate  where  the 
information  is  located  in  the  Student  Handout. 

Encouraged  instructors  through  lesson  plan  marginal  data 
to  routinely  announce  the  location  of  transitions  and 
main  points  in  the  Student  Handout. 

Produced  the  Student  Handout  outline  for  note  taking  so 
that  it  can  be  separated  from  the  other  student  material. 


b.  Was  there  sufficient  white  space  for  taking  notes? 

Specific  Comments : 

Yes,  there  was  plenty  of  room  to  write  in. 

There  was  more  white  space  than  needed.  Less  space  would 
do . 

Conclusion:  Reduced  the  requirement  and  the  amount  of  space 
for  note  taking. 

c.  Should  any  other  items  be  added? 

Specific  Comments: 

There  is  too  much  information  to  listen  and  write. 

Student  Handout  needs  more  information  bullets, 
especially  for  points  that  need  to  be  emphasized. 

We  should  have  more  in  information  in  front  of  us.  Then 
we  wouldn't  need  to  take  so  many  notes  and  could  listen 
more . 

Emphasize  note  taking.  If  everything  is  written  down, 
then  maybe  we  would  only  be  reading  the  material  instead 
of  paying  attention. 

Include  the  crosswalk  chart. 

Conclusion:  Added  details  on  main  points  and  key  talking 
points  from  the  Instructor's  Guide  and  included  the  crosswalk 
chart . 


11 .  Did  you  read  any  of  the  articles  in  the  Reference  Book? 

If  yes,  which  ones  and  were  they  informative? 

Summary:  Mixed  response. 

Specific  Comments : 

Good  reference  material.  Helpful. 

The  Navy  flight  surgeon  article  on  stress,  the  male- 
female  differences  in  communication  article,  and  MG 
Robinson's  article  were  especially  good. 

Read  all  of  them  but  don't  remember  them  well. 
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Case  studies  and  Flightfax  readings  were  more 
interesting. 

Some  were  scanned,  but  not  all.  My  night  flying  schedule 
was  too  demanding. 

Only  read  the  articles  referenced  in  class.  Didn't  have 
enough  time  to  read  any  other  articles. 

The  more  relevant  the  article  is  to  the  class,  the  better 
chance  there  is  that  I  would  read  it. 

Conclusions : 

Reviewed  all  articles  in  the  Background  Reading  section 
to  ensure  relevancy,  interest,  and  brevity. 

Included  a  brief  abstract  for  each  article  in  the  table 
of  contents. 

Added  reading  assignments  for  discussion  in  the  next 
day's  class  to  the  Trainer  and  Instructor's  Guides. 


12.  Was  the  mission  planning  and  rehearsal  practical  exercise 
helpful?  How  could  it  be  improved? 

Summary:  Mixed,  opinion. 

Specific  Comments: 

It  was  good;  a  bit  frustrating,  but  good.  It's  hard  to 
put  all  those  things  in  sequence.  [Majority  considered 
35  items  too  many.] 

Hard  to  do  in  a  group  because  of  different  opinions. 
[Others  found  group  discussion  to  be  useful  in 
discovering  rationale  for  priorities.] 

Need  to  allow  more  discussion  of  different  views  on 
priorities--this  caused  some  frustration  because  time  was 
not  allowed  to  form  a  consensus. 

Seemed  incomplete;  need  to  have  more  follow-up  on  why 
this  was  important. 

Have  to  allow  for  different  planning  sequences  for 
different  missions — no  right  answer  for  all  missions. 
[Groups  debated  the  need  for  a  "school  solution"  or  right 
answer . ] 

Need  a  model  of  basic  sequence  and  priorities  for  mission 
planning . 

Valuable  to  the  individual  to  think  about  priorities-- 
artificial  in  the  sense  that  individuals  usually  do  not 
plan  by  themselves;  they  plan  as  a  team. 

Could  be  improved  by  collapsing  the  35  items  into  10  to 
15  major  headings--planning  and  map  marking  can  be 
considered  the  same  thing. 

I  learned  what  all  the  planning  considerations  are. 
Usually,  being  pilot  of  a  flight,  I  only  do  a  part  of  the 
list.  Now  I  have  the  big  picture. 
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Not  needed  after  the  first  flight  with  a  crewmember, 
these  things  always  get  worked  out. 

Should  administer  the  exercise  to  crews  within  the  same 
unit  because  each  unit  plans  differently. 

Conclusion:  Consolidated  the  planning  items  and  recommended 

that  the  instructors  allow  no  more  than  2  crews  (4  to  6 
persons)  as  work  groups  during  the  exercise. 


13 .  Was  the  communications  practical  exercise  helpful? 

How  could  it  be  improved? 

Summary:  Mixed  opinion. 

Specific  Comments: 

Good  exercise,  but  need  a  more  tactical  example. 
Emphasize  the  use  of  standard  terminology  in  the 
exercise . 

Include  tactical  examples  like  LZ,  PZ,  BP,  and  firing 
positions . 

Require  a  description  of  something  tactical  with  no 
communications  feedback. 

Use  the  basic  figure,  then  go  on  to  a  tactical  example. 

Conclusion:  Included  tactical  examples  for  instructors  to 

supplement  the  basic  geometric  figure. 


14 .  Did  you  conplete  the  Hazardous  Thought  Pattern  exercise? 

If  not,  do  you  plan  to  do  so?  If  you  did,  was  it  helpful? 

Summary:  Mixed  opinion. 

Specific  Comments: 

Few  individuals  did  the  exercise. 

Many  disliked  the  forced  choice  nature  of  the  test-- 
difficult  to  select  a  response. 

Thought  it  was  very  good,  useful,  and  should  be  part  of 
the  course. 

Consider  using  the  Hazardous  Attitude  Survey. 

Conclusion:  Reworded  directions  to  increase  respondent 
comfort  with  the  forced  choice  nature  of  the  exercise  and 
included  notes  in  the  Instructor's  Guide  to  encourage  its  use 
during  the  course. 
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15.  Did  you  cort^jlete  the  Stress  Management  exercise? 

If  not,  do  you  plan  to  do  so?  If  you  did,  was  it  helpful? 

Summary:  No  opinion. 

Specific  Comments: 

Few  completed  the  exercise. 

One  crewmember  indicated  that  he  planned  to  complete  it. 

Conclusion:  Added  the  stress  management  exercise  as  an 
optional  in-class  practical  exercise  and  included  it  in  the 
outside  reading  assignments. 


16.  Was  the  course  the  right  length  to  teach  crew  coordination? 

If  not,  what  adjustments  are  necessary? 

Summary:  Mixed  opinion.  Generally  favorable. 

Specific  Comments : 

Course  is  about  the  right  length  for  initial  training. 
Consider  adding  another  day  and  conducting  the  classroom 
training  half-days  in  the  morning  only. 

Introduction  could  be  shortened  to  two  hours  since  most 
of  the  material  was  repeated  later  on. 

The  course  seemed  too  long  for  a  crew  who  had  flown 
together  a  long  time.  [Other  crews  debated  this  where 
there  was  not  familiarity.] 

Reduce  the  number  of  hours  if  possible.  Parts  seemed 
repetitive.  Condense  some  discussion. 

Conclusion:  Provided  options  in  the  Instructor's  Guide  for 

half-day  scheduling  of  the  classroom  instruction  phase. 
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Ill.  Flight  Simulator  (hands  on) 

1 .  Was  the  purpose  of  the  simulator  phase  explained? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

Yes.  Completely  understandable. 

It's  an  important  part  of  the  course,  so  I  knew  what  it 
was  for.  Makes  perfect  sense. 

It  was  clear  that  our  end  of  course  evaluation  was  the 
final  simulator  mission. 


2 .  Was  the  hands-on  phase  necessary  to  effectively  teach  crew 
coordination  principles? 

Yes.  Strong  agreement. 

Specific  Comments : 

Absolutely  essential. 

Definitely  needed  as  part  of  the  course.  Helps  reinforce 
classroom  instruction. 

New  techniques  should  be  practiced  in  the  safety  of  the 
simulator . 

Necessary  demonstration  of  crew  coordination  principles. 
Puts  principles  into  action.  This  way  I  really  learned. 
The  simulator  is  valuable,  but  training  in  an  actual 
aircraft  would  allow  crew  chief  participation. 

You  don't  worry  about  some  problems  as  much  in  the 
simulator,  as  compared  to  flying  the  actual  aircraft. 
Pre-training  evaluation  mission  may  be  negatively  biased 
because  some  of  the  crews  were  not  the  regular  battle- 
rostered  crews  [performance  in  aircraft  would  have  been 
better] . 

Conclusion:  Provided  options  in  the  Instructor's  Guide  for 
an  aircraft-based  training  or  evaluation  mission. 


3 .  Was  the  "crawl -walk-run"  approach  to  the  training  and 
evaluation  rides  effective?  More  rides  needed?  Adequate 
number  of  rides?  Too  many  rides? 

Summary:  Mixed  opinion  on  the  crawl-walk- run  approach  and 
the  number  of  missions. 

Specific  Comments : 

Pre-training  evaluation  mission  was  important  to  get  our 
attention,  helped  us  see  the  value  of  the  course,  and 
motivated  us  to  improve. 
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The  baseline  evaluation  mission  shouldn't  be  too 
difficult  or  it  can  create  hard  feelings.  I  didn't 
appreciate  the  19°  magnetic  variation  surprise. 

I  didn't  notice  much  difference  in  the  way  the  instructor 
interacted  during  the  simulator  missions.  I  don't  think 
our  crew  really  got  the  crawl-walk-run  training  approach. 
Instructors  should  be  more  aggressive  in  pointing  out 
errors  and  weaknesses. 

Training  rides  were  too  check-ride  oriented  with  too 
little  emphasis  on  crew  coordination  principles. 

The  number  of  simulator  missions  was  "just  right"  to 
provide  a  good  foundation. 

Need  more  simulator  missions.  We  continued  to  improve 
with  each  simulator  flight — more  is  better  within  time 
constraints . 

Not  enough  missions  to  obtain  CRLl . 

It  would  help  to  have  a  pre-training  practice  tactical 
mission.  We  usually  don't  fly  tactics  in  the  simulator. 
Keep  pre-training  evaluation  for  comparison  with  training 
missions  and  the  end  of  course  evaluation  mission. 

Conclusions : 

Expanded  the  Instructor  Guide  discussion  of  instructor 
and  evaluator  actions  during  simulator  or  flight  missions 
(i.e.,  emphasized  the  learning  and  not  the  evaluation 
aspects  of  the  training  missions) . 

Added  the  pre-training  evaluation  mission  to  the  course. 


4.  Did  you  have  enough  time  during  the  hands-on  periods;  that 
is,  pre-mission  planning  and  rehearsal,  mission  execution, 
and  after-action  review? 

Summary:  Mixed  opinion. 

Specific  Comments : 

Not  enough  time  for  the  pre-training  evaluation  mission. 
We  didn't  know  what  to  expect. 

There  was  adequate  time  after  we  had  the  classroom 
instruction  as  we  became  more  efficient  at  planning. 
Insufficient  time  to  plan  and  rehearse  such  a  complex 
mission.  It  would  take  a  day  to  plan  these  types  of 
missions  in  the  unit. 

There  is  no  need  for  IPs  to  read  the  OPORD  to  the  crew. 
Instructor  or  evaluator  should  provide  an  operations 
officer  mission  overview  and  be  available  to  answer 
questions . 

Some  aspects  of  the  mission  briefing,  like  weather,  were 
a  waste  of  time.  Better  to  just  give  the  crew  a  copy  of 
the  briefing  and  be  available  to  answer  questions. 
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Conclusion:  Emphasized  the  instructor  and  evaluator's  role 
as  operations  officer  and  use  of  the  air  mission  briefing  in 
the  Instructor  Guide. 


5.  You  participated  in  four  simulator  rides  during  the  testbed. 
What  did  you  think  about  these  rides  in  terms  of  whether  they 
reflect  the  types  of  missions  you  fly?  In  terms  of  the  level 
of  difficulty? 

Sunmary:  General  agreement  in  mission  realism  and 
difficulty. 

Specific  Comments: 

Scenarios  were  realistic  and  representative  of  daytime 
METL. 

Covered  unit  missions  except  night  conditions. 

Missions  were  challenging,  but  about  right.  Crew 
coordination  skills  improved  during  the  course  and  this 
increased  confidence. 

Consider  adding  a  NVD  mission  in  the  aircraft  or 
simulator.  Prefer  use  of  the  aircraft  due  to 
insufficient  light  contrast  in  the  simulator. 

Consider  having  crews  locate  themselves  after  being 
placed  on  random  terrain. 


Conclusion:  Included  an  option  in  the  Instructor's  Guide  to 

provide  for  an  additional  training  mission  to  address  NVD, 
actual  aircraft,  and/or  mixing  crewmembers. 


6.  Do  you  feel  that  the  use  of  the  videotape  of  your  crew's 

simulator  flight  during  the  instructor  debriefing  was  a  good 
training  technique?  Why? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

Yes,  absolutely  necessary. 

I  probably  learned  more  from  the  videotape  review. 
Feedback  on  actual  performance  is  the  best  part  of  the 
course. 

Helps  you  remember  what  happened,  check  problem  areas, 
and  settle  disagreements. 

Allows  you  to  recognize  your  own  weaknesses  and 
strengths,  recognize  mind  lapses,  identify  own  mistakes, 
review  own  performance,  and  tie  the  training  together. 
Need  tape  counters  and  mission  time  marks  to  locate 
actions  on  the  videotape  for  review.  [These  features 
were  present  in  the  testbed  equipment.  The  aviator  wants 
them  generally  available.] 
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Conclusion:  Emphasized  in  the  Instructor's  Guide  the 
importance  of  reviewing  videotapes  during  mission 
debriefings . 
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IV.  General  Observations 

1 .  What  is  your  overall  iiipression  of  the  adequacy  of  the 
aircrew  coordination  training  provided?  Do  you  have  any 
recommendations  for  iirprovement? 

Summary:  Course  is  fully  adequate.  General  agreement. 

Specific  Comments: 

It  was  an  excellent  course.  Glad  to  see  the  emphasis  on 
safety. 

I'm  100%  for  this  program.  It  will  save  lives  and 
aircraft.  It  may  well  save  my  life  one  day. 

I  thought  some  of  the  material  was  common  sense  but  it  is 
needed  to  institutionalize  common  practices  and  define 
techniques.  Makes  everyone  speak  up  and  fly  as  a  crew. 
Some  individuals  and  crews  were  already  practicing  crew 
coordination  skills  but  didn't  realize  it.  Now  we  knov: 
what  to  add  and  where  to  improve. 

Should  have  non-rated  crewmembers  attend  the  training. 

It  would  improve  their  assertiveness  and  acceptance  by 
rated  crewmembers.  Crews  would  communicate  better  and 
emphasize  the  importance  of  the  total  crew. 

Consider  scheduling  back-to-back  simulator  missions 
instead  of  the  day  or  two  break  between  simulator 
periods . 

Explore  the  benefit  of  interspersing  and/or  alternating 
academics  and  simulator  sessions. 

I  would  have  liked  to  be  able  to  keep  the  tapes  and 
review  them  when  I  had  more  time  at  home. 

I  felt  that  this  was  a  crash  course.  I  need  more  time  to 
try  it  out,  reflect,  and  discuss  it  with  IPs. 

Orient  the  course  for  unfamiliar  crews  rather  than  using 
it  to  progress  aviators  to  CRL-1  level. 

Videotapes  are  a  must  for  this  training.  Cameras  should 
be  installed  in  simulators  for  crew  coordination 
training. 

Conclusions : 

Emphasized  the  importance  of  including  non-rated 
crewmembers  in  the  course. 

Recommended  that  all  flight  simulators  and  designated 
aircraft  be  equipped  with  video  cameras  to  support  crew 
coordination  training. 


2 .  What  is  your  overall  irt^ression  of  the  conduct  of  the  aircrew 
coordination  evaluations?  Do  you  have  any  recommendations 
for  iirprovement? 

Summary:  Evaluations  were  consistent  with  current  practices. 

General  agreement. 
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Specific  Comments : 

It  is  good  to  evaluate  the  crew  instead  of  an  individual 
aviator  flying  with  an  IP. 

Evaluations  conducted  from  the  jump  seat  were  effective. 
Evaluators  in  the  role  of  crewmember  must  be  considered 
part  of  the  crew  workload. 

It  is  beneficial  when  IPs  instruct  as  they  evaluate. 
Simulator  console  operator  communications  can  lead  to 
crew  error,  especially  during  instrument  conditions. 

Need  to  place  more  emphasis  on  reviewing  the  principles 
taught  in  the  classroom  instead  of  emphasizing 
evaluation. 

Evaluation  debriefs  tended  to  be  checkride  oriented  at 
first;  then  they  became  more  instructional. 

Consider  conducting  one  evaluation  in  the  actual 
aircraft . 

Evaluate  unit  continuation  training,  too. 

Conclusions : 

Emphasized  the  instructional  nature  of  evaluations  and 
debriefings . 

Emphasized  the  liberal  use  of  unit  SOP  terms  (e.g.,  key 
words  to  prevent  gross  misorientation  during  terrain 
flight  navigation) . 


3 .  Do  you  personally  feel  that  you  are  now  better  prepared  to 
perform  as  an  Anty  aviation  crewmember?  Why? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments : 

I'm  a  lot  less  intimidated.  I  realize  that  I  have  a  lot 
to  contribute  as  a  crewmember. 

Made  me  think  more,  even  with  crewmembers  that  I  have 
flown  with  before. 

Helps  assertiveness  among  crewmembers.  Communication  is 
better . 

Have  used  crew  coordination  principles  already  on  the 
flight  line.  Crew  briefings  are  greatly  improved. 

Helped  with  workload  management  and  workload  awareness. 
Increased  awareness  of  the  need  for  open  communication  in 
the  cockpit. 

I  sequence  and  time  activities  better.  Got  me  out  of 
some  bad  habits. 

More  comfortable  in  the  cockpit.  Feel  like  more  of  a 
team  member. 

Much  of  the  course  material  was  subconscious  before,  but 
now  I  understand  why  I  do  things  better. 

I  know  what  I  do  well  and  not  so  well,  and  divide 
workload  with  other  crewmembers  accordingly. 
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Conclusion:  Emphasized  that  the  course  implements  the  Army's 
philosophical  shift  from  individual  aviators  to  crews  and 
that  students  should  expect  to  be  better  crewmembers  after 
the  training. 


4 .  Would  you  recommend  that  this  course  be  attended  by  every 
Army  aviator?  Why? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

Should  include  non-rated  crewmembers;  for  example,  crew 
chiefs,  door  gunners,  ground  crews  to  standardize  the 
entire  crew. 

Include  ATC  personnel. 

Integrate  the  course  into  lERW  and  other  institutional 
training. 

Conclusions : 

Emphasized  that  course  attendees  include  non-rated 
crewmembers . 

Recommended  crew  coordination  familiarization  training 
for  ground  crews. 


5.  If  this  training  package  is  fielded  Amy-wide,  should  there 
be  a  system  to  track  testbed  participants  in  longitudinal 
studies  of  the  Amy's  crew  coordination  program? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

Could  be  used  to  refine-  the  crew  coordination  program  and 
improve  the  training  course. 

Track  to  determine  the  length  of  time  needed  to  achieve 
superior  performance  level. 

Conclusion:  Recommended  that  the  class  roster  of  testbed 
participant  instructors  and  aircrews  be  retained  to  follow  up 
crew  coordination  policy,  program,  and  training  progress  over 
time . 


6.  What  effect  has  participation  in  this  project  had  on  you 
personally? 

Summary:  General  agreement  on  positive  effect. 

Specific  Comments : 

Communication  has  improved.  Improved  the  way  I  brief  and 
discuss  missions. 

Changed  mindset  from  single  pilot  to  team  concept. 
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Increased  confidence  in  self  and  crew. 

Understand  the  importance  of  planning. 

Course  will  probably  save  my  life  in  the  future, 
particularly  the  2-challenge  rule. 

Changed  my  mind  about  the  worth  of  simulator  missions, 
especially  the  need  to  brief  and  rehearse  within 
available  time. 

Raised  awareness  level  of  what's  going  on  in  the  cockpit. 
Caused  serious  evaluation  of  own  habits  and  self  as  a 
crewmember. 

More  detailed  thinking.  More  professional  in  speech  and 
actions. 

Able  to  sort  and  manage  tasks. 

Ask  more  questions  and  get  more  involved  in  flight. 

-  ,  Changed  from  cockpit  dictator  to  crewmember. 

Applied  crew  coordination  principles  to  other  courses  and 
at  home. 

More  open  in  the  cockpit.  Don't  assume  as  much. 

Conclusion:  Recommended  that  the  letter  of  transmittal  that 

accompanies  the  exportable  training  course  to  units  in  the 
field  convey  the  positive  effect. the  course  had  on  testbed 
participants . 


7 .  Do  you  have  any  questions  or  concerns  that  you  would  like  to 
ask  or  convey  to  the  crew  coordination  project  staff? 

Summary:  Open  discussion  surfaced  both  questions  and 
concerns . 

Specific  Comments: 

When  will  the  training  package  be  fielded?  1993. 

What  will  the  training  program  cost  the  Army?  [Estimate 
of  $4M.]  We  could  save  one  aircraft  and  this  course 
would  more  than  pay  for  itself. 

Battle  rostering  and  crew  coordination  should  be 
considered  separately. 

Introduce  more  variability  into  the  missions  to  avoid 
activity  patterns  like  IMC  always  the  last  segment. 


D-24 


Evaluator  and  Trainer  Exit  Interview 


EVALUATOR  AND  TRAINER  EXIT  INTERVIEW 


I .  Course  of  Instruction 

1 .  Has  adequate  time  (or  too  much  time)  been  allocated  for  each 
segment  of  the  course?  In  answering  this  question,  consider 
both  the  Trainer  Course  and  the  Student  Course. 

Summary:  General  agreement  that  the  amount  of  time  for  the 

Trainer  Course  and  the  Student  Course  is  adequate  and  that 
alternative  scheduling  options  and  command  emphasis  are 
needed  for  both  courses. 

Specific  Comments : 

I  think  that  there  was  adequate  time — don't  go  with  any 
less.  Class  effectiveness  and  use  of  class  time  is 
dependent  on  how  much  interaction  there  is  within  the 
group. 

If  time  constraints  are  not  a  big  problem,  the  course 
academics  should  run  four  hours  each  morning.  That  way, 
the  crews  can  do  their  required  work  in  the  afternoon  and 
instructors  can  prepare  the  next  day's  classes.  The 
half-day  schedule  would  be  preferable  for  commanders. 

Full  days  or  TDY  would  work  best.  If  we  schedule  only 
half  days,  the  crews  will  continue  to  have  additional 
duties  and  be  expected  to  fly  at  night.  Crews  should  be 
scheduled  so  there  are  no  conflicts. 

Put  the  course  on  the  training  schedule  so  that  the 
company  is  committed  for  three  days.  To  do  this,  the 
emphasis  has  to  come  from  the  brigade  or  the  division. 
Commanders  should  be  the  first  to  attend  the  course  to 
gain  their  support.  Positive  command  emphasis  would  make 
the  instructor's  job  easier. 

Need  more  classroom  time  to  cover  scenario  development 
including  information  on  10  scripts. 

Conclusions : 

Included  full-day  and  half-day  training  schedule  options 
in  the  course. 

Recommended  that  units  use  the  full-day  training  option 
with  students  placed  on  excused  duty  status.  If  excused 
duty  status  is  not  possible,  then  recommended  the  half¬ 
day  training  option. 

Recommended  that  crew  coordination  -training  be  placed  on 
the  training  schedule  with  the  same  command  emphasis  as 
field  training  exercises. 

Introduced  evaluation  and  scenario  development  activities 
in  the  Trainer  Course  to  cover  lessons  learned  and  10 
script  information. 
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2 .  Was  the  nvunber  of  video  segment  case  studies  adequate  to 
teach  the  13  Basic  Qualities? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

Yes.  I  thought  they  were  very  effective.  I  don't  think 
any  more  are  needed.  You  could  easily  pull  the  Basic 
Qualities  from  the  segments  with  a  little  thought. 

The  case  studies  and  video  segments  covered  more  than  one 
BQ  but  it  was  easy  for  the  crews  to  see  how  the  BQs  fit 
in.  They  were  good  choices. 

Video  segments  thoroughly  enhanced  class  participation. 
The  more  the  better. 

Need  a  video  segment  for  after  action  review  (AAR) . 

Strive  to  include  videos  that  represent  all  platforms. 

The  UH-1  rapelling  accident  was  very  effective. 

See  related  responses  at  item  I,  3. 

Conclusion:  Recommended  specific  after-action  review 

segments  from  the  testbed  videotapes  that  have  crewmember's 
consent  for  addition  to  the  course. 


3  .  Were  the  written  case  studies  effective  in  etnphasizing 
teaching  points?  If  yes,  was  the  number  of  wri.tten  case 
studies  adequate  to  teach  the  13  Basic  Qualities? 

Summary:  Yes.  General  agreement. 

Specific  Comments : 

Yes.  They  were  very  effective.  Videotapes  are  better 
but  the  written  case  studies  worked  too. 

Instructors  end  up  using  both  the  videos  and  the  written 
case  studies  together. 

Written  case  studies  allow  the  instructor  to  draw  out 
both  positive  and  negative  examples  during  discussion. 
Yes.  Each  case  study  covered  more  than  one  Basic  Quality 
and  it  was  easy  for  instructors  to  tie  them  together. 

The  cases  were  good  examples. 

There  are  definitely  enough  cases  to  teach  the  course. 
Instructors  should  add  their  own  cases  from  Flightfax  and 
other  sources  to  keep  the  course  current. 

Consider  adding  a  short  introduction  to  the  case  studies 
so  that  the  students  will  know  what  to  look  for  and  what 
went  wrong.  I  think  the  aviators  need  to  get  more  of  an 
idea  of  the  case  study's  focus. 

Need  a  written  case  study  for  AAR. 
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Conclusions : 

Recommended  the  use  of  the  UH-60  Fast  Rope  case  study  and 
video  segment  as  the  example  for  teaching  the  AAR 
instructional  segment. 

Emphasized  the  instructor's  responsibility  to  maintain 
the  course  by  adding  up-to-date  case  study  materials. 

4 .  Has  the  course  tied  crew  coordination  principles  and 
techniques  adequately  to  the  ATMs? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

Yes.  It  was  done  very  well.  The  ATM  standards  state, 
"correctly  perform  Crew  Coordination."  With  this  course, 
you  know  how  to  correctly  perform  the  crew  coordination 
component  of  the  ATM  tasks. 

The  relationships  really  came  together  after  two 
simulator  missions. 

I  am  concerned  that  TC  1-210  and  TC  1-212  will  be 
implemented  without  the  benefit  of  this  training.  The 
Crew  Coordination  program  should  become  official  in  order 
to  effectively  implement  the  TCs . 

See  related  responses  at  items  I,  2  and  3  and  II,  5. 

Conclusion:  The  course  is  effective  and  necessary  to 
implement  the  new  training  circulars. 


5 .  How  many  simulator  sessions  are  required  to  achieve  crew 
coordination  proficiency? 

Summary:  General  agreement  on  four  to  five  simulator 

missions  to  implement  the  Army's  crawl-walk-run  training 
philosophy. 

Specific  Comments : 

I  think  that  four  missions  are  necessary.  The  first  two 
missions  should  be  assisted,  then  the  third  should  be 
hands-off.  I  believe  this  would  impTement  the  crawl- 
walk-run  approach  and  result  in  improved  evaluation 
rides . 

It  should  be  four  with  three  training  (two  assisted,  one 
hands-off)  and  then  an  evaluation. 

If  I  really  had  my  preference,  we  would  have  five 
missions  and  include  the  baseline  for  teaching  purposes. 
Minimum  of  three,  maximum  of  five. 

Five  missions  as  follows: 

•  baseline  evaluation. 

•  three  training  (crawl,  walk,  run) 

•  final  evaluation. 
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-  ,  Third  mission  should  be  under  more  stressful  NVD 

conditions.  [Some  disagreement  that  this  might  be  too 
hard. ] 

Use  the  same  trainer  for  training  missions;  switch 
evaluators  from  baseline  to  final  evaluations. 

Don't  forget  crawl-walk-run.  If  training  is  too  rushed, 
nothing  will  be  reinforced.  Two  training  flights  are 
very  important  to  the  understanding  and  reinforcement  of 
the  classroom  instruction. 

Concl usions : 

Expanded  the  course  to  include  a  pre-training  mission  and 
a  course  completion  evaluation  mission  in  the  simulator 
or  aircraft. 

Modified  the  course  to  allow  for  local  consideration  of 
an  additional  training  mission  with  options  for  NVD, 
aircraft  instead  of  simulator,  and/or  mixing  crewmembers. 


6.  What  effect,  if  any,  did  the  pre- training  evaluation  mission 
in  the  simulator  have  on  the  classroom  instruction  part  of 
the  course?  Should  each  crew  be  required  to  complete  a  "crew 
coordination"  simulator  session  prior  to  the  classroom 
instruction? 

Summary:  General  agreement  that  the  pre-training  evaluation 

mission  enhances  classroom  effectiveness  and  should  be 
included  in  the  course. 

Specific  Comments: 

A  simulator  mission  should  precede  the  academics. 

It  imparts  the  proper  mindset  for  the  course  by  showing 
how  much  a  crew  doesn't  know  or  has  forgotten. 

A  great  effect. 

Yes.  This  needs  to  be  a  part  of  the  package!  It  was  so 
much  easier  to  teach  the  class  when  I  could  point  to  a 
crews'  performance  just  a  couple  of  days  before. 

It  brought  the  realism  home  to  them  because  they  could 
relate  the  classroom  environment  directly  to  themselves. 
Yes,  it  enhanced  class  participation. 

It  helps  to  be  able  to  tie  teaching  points  in  the  course 
to  mistakes  on  the  evaluation  mission. 

Yes,  the  simulator  mission  emphasizes  the  importance  of 
crew  coordination. 

No  effect.  The  simulator  missions  after  the  classroom 
training  were  effective. 

A  good  idea.  This  gave  the  students  a  before  and  after 
look . 

The  baseline  evaluation  ride  facilitates  classroom 
teaching  because  the  instructor  can  refer  back  to  it  for 
teaching  points. 
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Conclusion :  Expanded  the  course  to  include  a  pre-training 

mission. 


7.  Would  read-ahead  packages  enhance  the  flow  of  the  course?  If 
so,  what  material  would  you  suggest  be  included? 

Summary:  Yes.  General  agreement  on  read-ahead  package 

contents . 

Specific  Comments : 

I  think  a  general  synopsis  read- ahead  would  be  good. 
Consider  including  information  on  course  origin,  accident 
research  base,  history,  and  why  the  Army  is  going  in  this 
direction. 

Read-aheads  would  be  good  for  both  IPs  and  aviators. 

Could  possibly  reduce  the  amount  of  introductory  material 
by  synopsizing  the  read-ahead  in  the  first  part  of  the 
course  and  get  going  quicker. 

A  read-ahead  would  make  the  first  part  of  the  course 
easier  to  teach. 

Yes,  it  would  be  good  ideally,  but  I'm  not  sure  a  read- 
ahead  would  actually  be  used.  Aviators  will  take  it  but 
probably  won't  use  it.  To  be  read,  it  must  be  short. 
Include  topics  like  history  and  the  program's  origin,  the 
model,  objectives,  elements,  and  the  Basic  Qualities  to 
get  them  oriented. 

I  think  the  crosswalk  table  would  be  helpful. 

No,  read-aheads  are  not  usually  read. 

Could  detract  from  the  pre-evaluation  mission. 

Include  chapter  6  of  the  aircraft  ATM,  objectives,  Basic 
Qualities,  and  behavioral  anchors. 

Package  the  read-aheads  like  Flightfax  and  case  study 
type  articles. 

Issue  after  the  baseline  ride  but  before  the  classroom 
instruction. 

Include  the  history,  model,  CCO,  CCE,  BQ's,  with  very 
basic  definitions  and  tie  them  together  with  a  brief 
overview  of  the  entire  course.  Would  not  get  detailed  on 
any  one  subject. 

A  few  pages  introducing  concepts  and  techniques. 

To  be  honest  I  don't  know  if  the  student  would  use  them. 
Don't  eliminate  topics  from  the  course  because  they  are 
included  in  the  read- ahead. 

Conclusions : 

Included  read-ahead  (3-4  pages)  for  instructors  that 
include  case  study  examples  and  the  pre-training  reading 
assignments . 

Provided  two  student  read- ahead  packages.  One  is 
designed  for  use  before  the  pre-training  mission  and  the 
other  is  for  use  after  the  pre-training  mission. 
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8.  Overall,  did  the  Trainer  Course  adequately  prepare  you  to 
teach  the  aviator  Student  Course? 

Summary:  Yes.  General  agreement. 

Specific  Comments: 

Yes,  especially  when  supplemented  with  knowledge  gained 
from  a  broad  base  of  operational  experience. 

Inexperienced  UTs  need  to  rehearse  with  IPs  to  be 
completely  comfortable  and  not  too  dependent  on  the 
Instructor  Course  materials  for  examples. 

I  think  the  more  you  teach  the  course  the  better  you  get. 
Now  that  I've  memorized  the  13  BQs  I'm  a  much  better 
instructor  and  evaluator. 

Yes,  the  exception  is  that  I  needed  more  time  to  prepare 
the  academic  instruction. 

The  lOs  should  monitor  the  course  to  understand  crew 
coordination  principles  and  evaluation  techniques. 

Would  have  liked  to  have  flown  an  evaluation  mission. 
Consider  substituting  a  baseline  evaluation  mission  for 
the  Trainer  Course  scenario  familiarization  mission. 

IPs  and  UTs  should  complete  an  evaluation  mission  during 
the  Trainer  Course. 

Conclusions : 

Recommended  that  instructors  need  at  least  2  hours  of 
preparation  time  for  every  1  hour  of  classroom 
instruction. 

Added  a  complete  scenario  familiarization  mission  to  the 
Trainer  Course  with  instructors  performing  both 
crewmember  and  evaluator  duties  critiqued  by  course 
trainers . 

Included  all  Student  Course  training  and  evaluation 
missions  in  the  Trainer  Course. 

9.  Are  there  any  Trainer  Course  segments  (for  exaitple,  MOI , 
evaluation,  scenario  familiarization)  that  should  receive 
more  or  less  emphasis? 

Summary:  Yes.  General  agreement  on  the  need  for  more 
emphasis  on  scenario  development. 

Specific  Comments: 

I  could  have  used  a  better  overview  of  the  evaluation 
scenarios  and  how  they  were  developed.  Consider  giving 
us  the  OPORD  and  the  evaluator  worksheet  as  a  reading 
assignment  prior  to  classroom  discussion. 

It  would  be  good  to  have  a  couple  of  standard  missions  in 
the  instructor's  package.  We  could  use  them  as  examples 
to  construct  our  training  scenarios. 

Need  to  emphasize  the  scenario  development  section. 
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Need  more  time  for  planning  at  the  simulator  facility  to 
develop  scenarios  to  support  training  and  evaluation 
missions . 

More  time  could  have  been  spent  on  evaluation  principles. 
Need  improved  examples  for  the  instructors  to  grade 
during  classroom  exercises  for  proficiency  when  the 
grading  counts.  [See  related  comments  at  item  IV,  8.] 
Continue  to  emphasize  the  course  objectives  and 
supporting  statistics  during  the  first  hour  of 
instruction. 

Consider  strengthening  the  MOI  refresher  segment  and 
adding  an  instructor  presentation  and  critique  period. 

Conclusions : 

Introduced  evaluation  and  scenario  development  activities 
in  the  Trainer  Course  to  include  scenario  development 
lessons  learned. 

Identified  candidate  video  segments  from  testbed  crews  to 
replace  the  Evaluator  Workshop  audio-only  examples  for 
practice  ratings.  Included  narratives  from  the 
identified  video  segments  in  the  Trainer  Course. 
Strengthened  the  MOI  refresher  segment. 
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II.  Scenarios  Used  During  the  Evaluation  Phase 

1 .  Were  the  evaluation  scenarios  of  about  the  correct  level  of 
difficulty? 

Summary:  Yes.  General  agreement. 

Specific  Comments : 

Yes.  Scenarios  were  well  thought  out,  well  developed, 
and  just  the  right  degree  of  difficulty. 

Yes.  They  covered  most  of  the  aspects  of  what  we  do. 
However,  we  need  to  tailor  some  scenarios  for  NVD.  Crew 
Coordination  is  much  more  important  in  a  more  stressful 
environment.  We  have  to  train  in  the  hardest  mode  which 
is  NVD. 

Consider  a  third  "walk"  training  ride  with  NVD.  Suggest 
a  day-NVD-day  mission  progression. 

Yes,  at  first  they  were  a  little  difficult  but  after 
training  the  crews  had  little  or  no  trouble  with  them. 
Scenarios  could  have  been  a  little  bit  more  difficult — 
more  "hard  location  times"  and  complex  routes. 

Conclusion:  Modified  the  Student  Course  to  allow  for  local 
consideration  of  a  fourth  training  mission  with  options  for 
NVD,  aircraft  instead  of  simulator,  and/or  mixing 
crewmembers . 


2 .  Was  the  scenario  reasonably  realistic  in  terms  of  the  types 
of  missions  these  aviators  must  execute? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

Yes.  Each  aviation  unit  will  need  guidance  on  how  to 
develop  scenarios. 

IPs  should  add  realism  by  supplementing  scripted  events 
and  providing  communications  to  remind  crews  that  they 
are  part  of  a  multi-ship  mission.  This  includes 
navigation  assistance  and  other  unit  SOP  actions. 

Need  to  include  nonrated  crew  members  in  the  training, 
especially  for  the  actual  aircraft  flight. 

See  related  comments  at  item  II,  7. 

Conclusion:  Emphasized  the  importance  of  the  scenario 
development  guidelines  section  in  the  Instructor  Course. 
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3.  Was  there  enough  pre-mission  planning  time  for  the  crews? 

Summary:  Yes.  General  agreement  that  as  training  progressed 
crews  went  from  being  pressed  for  time  to  having  more  than 
enough  time  for  pre-mission  planning  to  include  mental 
rehearsal , 

Specific  Comments : 

[At  testbed  start,  some  instructors  used  planning  time  to 
read  through  the  entire  OPORD  instead  of  briefing  only 
the  main  points.  This  practice  was  corrected.  See  item 
II,  4.] 

Initially,  the  crews  felt  pressed  for  time.  By  the  final 
evaluation  mission,  all  crews  had  time  to  spare. 

At  first,  the  crews  did  not  have  adequate  time.  However, 
after  the  third  mission,  they  had  time  left  over.  The 
time  is  just  right. 

Crew  coordination  instruction  had  a  significant  effect. 
Proficiency  improved  with  instruction  and  practice. 

Going  from  not  near  enough  time  to  time  left  over,  crews 
were  able  to  conduct  pre-mission  planning  and  rehearsal 
that  covered  everything  in  the  ATM. 

Crews  learned  to  do  mental  rehearsal  once  they  got  time- 
organized.  This  technique  [rehearsals]  was  learned  from 
the  course. 

Too  time  constrained  to  mentally  rehearse  at  first, 
better  time  management  skills  acquired  during  the  course 
permitted  some  mental  rehearsal. 

I  noticed  that  the  crews  were  doing  mental  rehearsal. 
Everybody  improved  dramatically.  Some  [crews]  rehearsed 
contingencies . 

Before  the  training,  crews  usually  wouldn't  mentally 
rehearse,  but  after  the  training,  I  saw  them  conduct 
mental  rehearsals.  Most  crews  were  talking  about  "what 
if"  regarding  the  threat,  the  corridors,  etc. 

Conclusions : 

Retained  one  and  one-half  hours  for  pre-mission  planning 
and  rehearsal  in  simulator  and  aircraft  crew  coordination 
training  missions. 

Emphasized  the  importance  of  mental  rehearsal,  before 
flight  and  in  the  cockpit,  in  the  course  of  instruction. 


4 .  Do  you  have  any  suggestions  for  irtproving  the  preflight  air 
mission  briefing  and  crew  mission  briefings? 

Summary:  General  agreement  on  presenting  the  mission 

briefing  as  efficiently  as  possible  to  preserve  planning  time 
for  the  crew. 
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Specific  Comments: 

Make  sure  that  the  instructors  brief  the  mission  and  not 
cut  into  the  crew's  available  planning  time  by  reading 
the  entire  OPORD.  The  crews  can  read  their  copy  of  the 
OPORD  for  coordinates,  frequencies,  etc. 

I  think  that  the  instructor's  briefing  should  be  an 
overview  with  highlights  of  the  most  important  parts  of 
the  mission. 

Instructors  should  act  as  an  operations  officer  during 
the  planning. 

Consider  briefing  by  the  flow  of  mission  segments  (e.g., 
ingress,  egress)  versus  air  mission  briefing  format 
(e.g.,  5  paragraph  field  order). 

Consider  evaluating  the  crew's  ability  to  derive  the 
appropriate  mission  essential  information  from  the  OPORD. 
The  most  important  thing  during  the  briefing  is  to  get 
the  correct  and  the  right  amount  of  information  to  the 
crews  as  quickly  as  possible.  I  think  we  accomplished 
that . 

Emphasize  the  1/3  :  2/3  planning  rule  to  ensure  that 
crews  have  at  least  2/3  of  the  total  time  available  for 
planning. 

Consider  retaining  the  capability  to  videotape  crew 
mission  planning  in  the  course. 

Crew  briefing  should  use  the  ATM  checklist. 

Conclusion:  Emphasized  that  instructors  role  play  the 

Operations  Officer  during  pre-mission  planning  and  rehearsal. 


5 .  Do  you  have  any  suggestions  for  in^jroving  the  post-flight 
crew-level  after-action  review? 

Summary:  Strong  agreement  on  the  need  to  develop  an  AAR 

[after-action  review]  checklist  for  the  course  and  an  AAR 
task  for  all  ATMs. 

Specific  Comments : 

Crews  didn't  fully  understand  crew-level  AAR  content. 
Their  AARs  tended  to  focus  exclusively  on  technical 
aspects  of  the  mission  with  little  or  no  mention  of  crew 
coordination  interactions  except  during  videotape  review 
of  their  flight. 

AAR  became  natural  over  time  and  all  crews  generally 
improved,  but  we  have  to  change  their  mindset. 

Need  a  sample  AAR  checklist  included  in  the  course  to 
orient  discussions  on  crew  coordination  issues. 

Emphasize  the  goals  and  objectives  of  AAR  (e.g.,  identify 
problems,  causes,  and  solutions  for  mission  improvement) 
along  with  instruction  on  how  to  conduct  an  AAR. 
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Develop  an  AAR  checklist  that  can  be  used  on  the 
aviator's  kneeboard. 

Need  an  AAR  task  for  all  aircraft  ATMs  that  addresses  the 
entire  mission  from  pre-mission  planning  through  AAR. 

Conclusions : 

Developed  and  included  an  AAR  checklist  and  discussion  on 
its  use  in  the  course  of  instruction. 

Recommended  development  of  a  Crew-Level  After-Action 
Review  Task  to  be  included  in  each  aircraft  ATM. 


6.  Do  you  have  any  suggestions  for  in^jroving  the  post-flight 
evaluator  debriefing? 

Summary:  Strong  agreement  on  the  suggested  format  and 

benefit  of  videotape  review  as  an  evaluator  debriefing 
technique . 

Specific  Comments : 

Videotape  review  is  a  great  technique!  Crews  really  were 
interested  in  reviewing  their  performance. 

It's  important  to  have  the  tapes  available  for  the 
evaluator's  personal  review. 

Used  the  evaluator  worksheets  as  reminders  and  cues  to 
videotape  events. 

No  improvements  needed.  Everything  was  available,  such 
as  tapes — which  was  the  big  thing. 

It  was  a  challenge  to  separate  the  evaluation  of  the 
technical  aspects  of  the  mission  and  the  crew 
coordination  aspects.  I  tried  to  address  both  but 
individual  evaluator  style  and  technique  will  be  an 
influence  until  the  new  ATM  tasks  are  fully  understood. 
Videotape  capability  is  a  must-have  for  this  course  to 
succeed. 

Conclusion :  Retained  the  evaluator  debriefing  format  and 

emphasized  the  use  of  videotape  review. 


7 .  What  general  comments  did  the  aircrews  make  that  might  help 
us  in^rove  the  scenario? 

Summary:  General  agreement  on  improvements  to  overcome  the 
"simulator  mindset"  that  hinders  execution  of  realistic 
tactical  scenarios. 

Specific  Comments: 

The  biggest  thing  is  the  "mindset"  that  aviators  have 
when  using  the  simulator.  It  takes  a  while  for  the  crews 
to  feel  that  it's  realistic. 


D-35 


Evaluator  and  Trainer  Exit  Interview 


Multiple-aircraft  operations  in  the  simulator  are 
difficult.  It's  too  "quiet"  in  the  simulator;  the 
aircraft  to  aircraft  interaction  was  not  present.  Aboard 
the  single  ship,  the  workload  was  distributed 
realistically. 

Include  more  scripted  items  on  interaction  with  other 
aircraft  (e.g.,  communication  checks,  other  aircraft 
status,  navigation  assistance) . 

Consider  allowing  the  10  to  represent  other  aircraft 
input  (e.g.,  navigation). 

Crews  enjoyed  the  tactical  missions. 

Navigation  corridors  were  not  too  restrictive. 

Make  available  all  unit  SOP  mission  planning  materials 
(e.g.,  kneeboard  size  air  mission  briefing  cards)  to 
summarize  scenario  segments  and  events.  [Provided  once 
the  need  was  recognized.] 

Crews  suggested  using  laminated  maps  for  direct  posting 
of  scenario  information.  Map  covers  in  the  simulator 
facility  flight  kits  allowed  maps  to  shift  underneath  the 
mission  planning  markup  causing  errors  in  scenario 
execution  (e.g.,  routes,  check  points,  PZs  and  LZs) . 

Conclusions : 

Expanded  the  course  to  include  a  third  training  mission 
in  the  simulator  or  aircraft. 

Emphasized  the  importance  of  the  scenario  development 
guidelines  section  in  the  Trainer  Course. 


8 .  Do  you  have  any  specific  suggestions  for  improving  the 
scenario  and/or  the  mission  objectives? 

Summary:  Yes.  General  agreement  on  including  NVD 
conditions . 

Specific  Comments : 

We  need  to  conduct  this  type  of  training  under  NVGs ! 
There's  a  drastic  difference  between  day  and  NVGs  [NVD 
conditions] . 

The  NVD  environment  would  in  itself  reinforce  the  concept 
of  proper  pre-mission  planning  and  crew  coordination 
throughout  the  mission. 

Make  one  of  the  three  training  missions  "with  NVG". 
Include  more  precise  mission  requirements  like  very 
specific  locations  and  timing  for  extraction  or  SAR 
(search  and  rescue) . 

With  the  equipment  we  have  available  now,  the  scenarios 
covered  all  mission  profiles  of  our  unit  mission.  Some 
change  to  the  missions  would  have  to  be  made  for  unique 
missions  (e.g.,  medevac,  electronic  helicopter). 
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Conclusion:  Modified  the  course  to  allow  for  local 
consideration  of  a  fourth  training  mission  with  options  for 
NVD,  aircraft  instead  of  simulator,  and/or  mixing 
crewmembers . 


9 .  Do  you  have  any  specific  suggestions  for  in^roving  the 
scenario  by  adding  or  deleting  tasks? 

Summary:  Yes.  General  agreement  on  substituting  manual 

tasks  for  system  based  tasks  when  system  capabilities  and 
procedures  are  in  transition  (e.g.,  electronic  aided 
navigation) . 

Specific  Comments: 

Remove  the  task  "electronic  aided  navigation"  unless  the 
equipment  and  procedures  in  the  simulator  are  the  same  as 
in  the  aircraft.  Unit  aircraft  have  doppler — the 
simulator  has  GPS. 

New  aviators  coming  out  of  flight  school  use  GPS  not 
doppler — aviators  in  the  units  use  doppler  and  they  don't 
know  how  to  use  GPS. 

Consider  adding  "dead  reckoning  navigation"  and  "FM 
homing"  tasks. 

The  scenario  covers  the  tasks  very  well.  I've  discovered 
that  some  crews  are  having  problems  with  the  most  basic 
tasks  (e.g.,  fuel  check). 

Emphasize  that  not  every  task  is  evaluated  in  every 
mission  segment. 

Conclusion :  Emphasized  the  importance  of  task  selection  in 

the  scenario  development  guidelines  section  of  the  Trainer 
Course . 


10.  Did  the  scenarios  allow  adequate  demonstration  and 

observation  of  the  13  Basic  Qualities  associated  with  crew 
coordination? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

Yes,  they  are  all  interconnected,  and  the  scenarios  gave 
us  an  opportunity  to  rate  all  of  the  BQs . 

I  thought  it  [demonstration  of  Basic  Qualities]  was  very 
good.  All  13  [Basic  Qualities]  were  there.  No  problems. 
All  [Basic  Qualities]  are  covered  in  the  scenarios. 

Expect  that  almost  any  mission  would  accommodate  all  of 
the  Basic  Qualities. 
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Conclusion:  No  change  to  scenario  development  guidelines. 
Crew  coordination  Basic  Qualities  will  be  demonstrated  in  any 
well  developed  mission  scenario. 


11.  If  the  answer  to  item  10  above  is  no,  what  tasks  or  mission 
events  should  be  added  to  permit  adequate  demonstration  and 
observation  of  each  Basic  Quality? 

Summary:  Does  not  apply. 
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III.  Evaluator's  Workbook 

1 .  How  did  you  utilize  the  Evaluator ' s  Workbook? 

Summary:  General  agreement  on  the  effectiveness  of  the 
Workbook  and  its  applicability  to  all  phases  of  a  mission. 

See  comments  and  conclusions  for  items  a  through  d. 

a.  During  pre-mission  activities? 

Specific  Comments: 

I  really  liked  this  book.  I  referred  to  the  schedule 
several  times  at  first  to  help  establish  my  own  schedule. 
I  used  the  OPORD  and  air  mission  briefing  to  brief  the 
crews  and  answer  questions  on  the  mission. 

Used  the  evaluator  worksheets  and  rating  guide  to  rate 
the  crews  during  pre-mission  planning. 

Reviewed  ratings  for  the  13  BQ's  to  determine 
performance . 

I  referred  several  times  to  the  rating  factors  but  mainly 
to  the  behavioral  anchors  to  determine  ratings  and 
grades . 

I  used  the  worksheets  extensively  and  filled  in  notes  as 
things  went  along. 

Conclusion:  Recommended  use  of  similar  scenario  materials, 
rating  guides,  and  worksheet  organization  to  support 
evaluation  of  pre-mission  planning  in  continuation  training. 

b .  During  the  flight? 

Specific  Comments: 

I  really  used  the  worksheets  a  lot.  They  helped  keep 
track  of  the  mission  and  maintain  the  flow  of  the 
evaluation  in  the  simulator.  The  ones  we  received  for 
the  testbed  evaluations  were  so  good  and  useful  that  we 
even  created  our  own  [based  on  the  testbed  version]  for 
the  training  missions. 

I  used  the  worksheets  systematically  to  rate  all  of  the 
BQs  task  by  task  and  segment  by  segment. 

With  the  worksheet,  I  really  1  didn't  have  to  use  the 
videotape  very  much.  The  worksheets  were  one  of  the  best 
evaluation  techniques. 

I  used  the  BQ  references  at  the  bottom  of  the  evaluator 
worksheets  to  relate  what  I  observed  during  the  mission 
to  the  rating  guide. 

The  rating  guide  evaluation  factors  helped  confirm  the 
correct  behavioral  anchors  for  final  grading. 
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The  notes  block  was  helpful  for  posting  notes  in  real 
time . 

Consider  a  kneeboard  size  version  of  the  evaluator 
worksheet  for  use  in  the  aircraft.  Keep  the  full  page 
version  for  use  in  the  simulator. 

Conclusion:  Recommended  use  of  the  worksheet  format  and 
rating  guides  for  continuation  training  evaluations  in  the 
simulator  and  aircraft. 

c.  During  post-mission  activities? 

Specific  Comments : 

I  used  the  behavioral  anchors  along  with  the  ATM  to  help 
crews  self  evaluate. 

Used  the  worksheets  to  make  notes  and  highlight  good  and 
bad  points  of  the  crew's  performance. 

The  worksheets  proved  to  be  an  ideal  way  to  review  the 
flight  with  the  crews  from  start  to  finish. 

I  reviewed  my  worksheet  notes  to  debrief  the  crews. 

[Some  evaluators  used  just  the  videotapes  for  mission 
review  and  crew  debriefings] . 

Conclusion:  Recommended  that  worksheets  for  continuation 

training  evaluations  include  space  for  note  taking  to  support 
crew  debriefings  and  grade  slip  comments. 

d.  After  the  mission  was  conpleted  (grading  and  rating)? 

Specific  Comments: 

I  used  the  worksheet  extensively. 

Used  the  rating  guide  to  help  complete  the  grade  slips. 
Reviewed  the  behavioral  anchors  to  determine  final  grades 
because  more  time  was  available  and  the  behavioral 
anchors  had  more  information  upon  which  to  base  a  grade. 
Used  the  guidance  provided  by  the  behavioral  anchors  not 
the  rating  factors  in  the  rating  guide. 

Conclusion:  Recommended  that  the  grading  and  rating  methods 
and  materials  used  in  ,the  testbed  be  modified  for  use  in 
continuation  training  evaluations. 


2 .  Regarding  the  videotapes? 

Summary:  Strong  agreement  that  crew  coordination  training 

absolutely  requires  videotape  capability.  Videotaping  is 
essential  to  demonstrate  teaching  points  and  provide  feedback 
to  crews.  Audio  only  is  not  acceptable.  See  comments  and 
conclusions  for  items  a  through  d. 
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a.  Did  you  review  then  evaluate ,  or  did  you  review  and 
evaluate  at  the  same  time? 

Specific  Comments: 

I  used  the  evaluator  worksheets  during  the  mission  and 
posted  final  grades  and  ratings  after  reviewing  the 
videotapes . 

The  review  and  evaluate  sequence  was  situationally 
dependent . 

I  did  not  follow  one  specific  procedure.  In  some  cases, 
it  was  better  to  just  let  the  crews  see  what  they  did  on 
their  own.  In  other  cases  the  crews  generally  knew  their 
strengths  and  weaknesses. 

The  technique  used  depended  on  the  crew.  For  a  quiet 
crew,  I  would  review  the  videotape  and  then  evaluate. 

If  my  worksheet  notes  weren't  specific  enough,  I 
evaluated  and  then  reviewed  the  videotape.  I  didn't  use 
the  videotapes  exclusively  because  the  worksheet  notes 
were  working  well. 

If  my  worksheet  notes  were  not  clear,  then  the  crew  and  I 
would  find  the  specific  point  on  the  tape  and  review  and 
evaluate  at  the  same  time. 

Conclusion:  Recommended  the  use  of  both  videotape  and 
evaluator  worksheets  for  continuation  training  evaluations. 


b.  Did  you  use  the  videotapes  to  review  specific  areas  of 
the  tape  where  you  thought  you  missed  iir5>ortant 
information? 

Specific  Comments : 

Yes.  Used  simulator  mission  time  to  annotate  worksheet 
as  a  cue  to  where  to  look  on  the  tape  if  I  had  a  question 
and/or  thought  that  I  had  missed  something. 

The  testbed  data  collection  "logger  sheet"  was  helpful  to 
locate  events  on  the  tape. 

I  used  the  videotapes  to  check  problem  areas  or  areas  of 
uncertainty. 

Major  benefit  is  to  clear  up  disagreements. 

Review  wasn't  always  necessary  if  I  recorded  everything 
on  the  evaluation  worksheets. 

Conclusion:  Provided  additional  guidance  on  the  use  of 

videotapes  to  debrief  crews. 

c.  Did  you  review  the  whole  tape? 

Specific  Comments: 

No.  Too  time  consuming. 

Never  the  entire  videotape;  either  with  the  crew  or 
alone . 
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Just  segments  based  on  evaluator  worksheet  notes  and 
selected  portions  to  debrief  the  crew. 

Reviewed  some  entire  segments  to  grade  tasks  like, 
conduct  fuel  check. 

Conclusion:  Provided  additional  guidance  on  the  use  of 

videotapes  to  debrief  crews. 

d.  What  general  comments  did  the  aircrews  make  as  they 
observed  their  videotape? 

Specific  Comments: 

Videotaping  is  an  excellent  training  aid  that  we  should 
have  all  of  the  time. 

They  were  able  to  critique  themselves  and  found  it  very 
helpful. 

•  "I  can't  believe  I  did  that!" 

•  Recognized  poor  use  of  time. 

•  Supported  self-AAR. 

•  Noticed  errors  like  rally  terms  versus  directional 
headings  in  degrees. 

One  crew  wanted  to  review  their  whole  tape  and  several 
crews  asked  if  they  could  keep  the  tapes  after  the 
training  rides. 

Didn't  understand  how  they  interacted  until  they  saw 
themselves  on  tape. 

•  "I  didn't  realize  I  did  that". 

•  "I  don't  remember  doing  that". 

Emergency  procedures  were  viewed  with  interest. 

For  some,  videotaping  was  kind  of  an  ego  bust.  Some 
realized  that  they  were  more  strict  or  authoritarian  than 
they  had  thought. 

The  crews  could  definitely  see  a  difference  in  themselves 
as  the  training  progressed  from  start  to  finish. 

They  could  actually  see  their  mistakes  as  someone  else 
would  see  them. 

We  need  to  have  the  ability  to  videotape  the  missions  so 
that  evaluators  and  crews  alike  can  review  and  learn  from 
their  performance  as  this  program  is  implemented  in  the 
units . 

No  crewmembers  felt  intimidated  by  the  video  cameras. 

They  quickly  forgot  that  the  cameras  were  even  there. 

Conclusion:  Recommended  that  all  simulators  and  designated 
aircraft  be  equipped  with  video  cameras  to  support  crew 
coordination  training. 
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3 .  What  elements  of  the  Evaluator  Worksheets  were  helpful  or  not 
helpful ,  and  why? 

Summary:  General  agreement  that  all  worksheet  elements  were 
helpful  in  minimizing  the  amount  of  time  that  evaluators 
spent  recording  their  observations  of  crew  performance. 

Specific  Comments:  I  used  the  entire  worksheet.  It  was  all 
useful  and  helpful.  See  additional  comments  at  items  a 
through  f. 

Conclusion:  Emphasized  the  role  of  evaluator  worksheets  and 
their  use  in  evaluator  workshops.  Recommended  evaluator 
worksheets  for  initial  and/or  refresher  and  continuation 
training  evaluations. 

a.  Segment  description? 

Specific  Comments : 

Allowed  us  [IPs]  to  concentrate  on  the  mission  not  10 
duties . 

[IPs  and  UTs  included  segment  description  information  in 
their  own  worksheets  developed  to  support  training 
scenarios . ] 

b.  ATM  Task? 

Specific  Comments : 

Provided  a  good  indexing  as  crews  progressed  through  the 
mission. 

Followed  the  sequence  of  scenario  activities. 

Provided  the  ATM  task  number  and  title. 

c.  ATM  Task  performance? 

Specific  Comments : 

Very  handy  to  be  able  to  mark  observations  quickly. 

I  could  initially  make  a  grade  then  review  it  later  using 
the  behavioral  anchors  for  a  final  grade. 

It  helped  me  grade  items  that  I  might  otherwise  not 
remember . 

d.  Segment  overall  performance? 

Specific  Comments : 

Segment  performance  criteria  was  too  single-task 
specific. 

It  helped  me  determine  overall  ratings  and  grades. 
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e.  Aircrew  Coordination  Basic  Qualities? 

Specific  Comments : 

It  was  especially  good  to  have  the  short  titles  at  the 
bottom  of  the  page  for  reference. 

Consider  including  blank  spaces  for  up  to  three  Basic 
Quality  entries  per  task. 

f.  Evaluator's  Notes? 

Specific  Comments : 

Plenty  of  space  for  notes  was  extremely  helpful  and  used 
routinely. 

Handy  to  record  initial  impressions  of  crew  performance, 
actions,  and  related  circumstances  such  as  simulator 
malfunctions,  etc. 


4 .  Were  there  elements  in  the  Workbook  that  were  confusing?  If 
so ,  which? 

Summary:  Mixed  opinion. 

Specific  Comments: 

No,  I  found  it  all  straightforward  and  easy  to 
understand. 

I  may  have  been  inconsistent  when  I  transitioned  from  the 
worksheet  to  the  grade  slip  to  grade  the  overall  flight. 

I  considered  the  grade  for  the  whole  mission  to  be 
different  from  the  sum  of  the  mission  segments. 

The  terms  "behavioral  anchors"  and  "behavioral  factors" 
were  confusing. 

See  related  comments  at  item  IV,  7. 

Conclusions : 

Emphasized  the  "whole  mission"  context  for  grading  the 
overall  flight  on  the  Crew  Evaluation/Training  Grade  Slip 
(DA  Form  7172-R) . 

Clarified  the  difference  and  use  of  effectiveness 
(rating)  factors  and  behavioral  anchors  in  the  Trainer 
Course . 


5 .  Are  there  some  elements  in  the  Workbook  that  you  could  do 
without?  If  so,  which? 

Summary:  Mixed  opinion. 

Specific  Comments : 

No,  I  liked  it  the  way  it  is  and  used  it  thoroughly. 
Remove  the  effectiveness  (rating)  factors  or  place  them 
behind  the  behavioral  anchors. 
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Remove  the  Unit  SOP  material. 

Generally,  I  only  used  the  evaluator  worksheets  and  the 
debriefing  guide. 

Conclusion:  Recommended  secondary  placement  of  the 

effectiveness  (rating)  factors  and  deletion  of  unit  SOP 
materials  from  the  continuation  training  evaluation 
materials . 


6 .  Are  there  any  additional  elements  that  you  would  like  to  add? 

Summary:  Mixed  opinion. 

Specific  Comments: 

No.  A  great  concept  that  was  of  help  to  me  in  evaluating 
the  crews. 

Add  an  10  worksheet  for  the  scenarios. 

Add  the  after-action  review  as  a  segment  on  the 
worksheets . 

Add  an  index  page  of  the  full  titles  for  the  Basic 
Qualities . 

Add  a  crew  mission  brief  and  AAR  form  (DA  Form  5484-R) . 
Add  a  kneeboard  size  mission  outline  or  mission  graphic 
for  reference  during  the  mission. 

Consider  adding  the  "Suggested  Points  to  Structure 
Debriefing  Questions"  from  the  Trainer  Course  materials. 

Conclusion:  Recommended  including  selected  additions  to  the 
materials  for  continuation  training  evaluations. 


7 .  Do  you  have  any  other  suggestions  for  iitproving  the  Workbook 
(be  specific) ? 

Summary:  General  agreement  on  the  need  for  an  evaluator 

worksheet  suitable  for  use  in  the  aircraft. 

Specific  Comments : 

Consider  an  abbreviated  form  of  the  worksheet  reducible 
to  kneeboard  size. 

Include  an  10  scenario  script. 

Consider  including  a  strip  map  of  mission  segments. 
Consider  moving  the  Basic  Quality  titles  toward  the  front 
of  the  workbook. 

Provide  evaluator  worksheets  to  UTs  for  use  in  initial 
and/or  refresher  training  missions. 
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Conclusions : 

Recommended  evaluator  worksheets  for  initial  and/or 
refresher  and  continuation  training  evaluations. 
Included  an  abbreviated  evaluator  worksheet  format 
reducible  to  grade  slip  size  for  use  on  aviator  knee 
boards  in  the  Exportable  Evaluation  Package  for  crew 
coordination  continuation  training. 
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Aircrew  Coordination  Basic  Qualities 

Can  any  of  the  13  Basic  Qualities  be  combined  or  eliminated? 

Summary:  At  first  IPs  thought  that  some  of  the  Basic 
Qualities  could  be  combined,  but  changed  their  minds  after 
working  with  them. 

Specific  Comments: 

[Some  IPs  suggested  that  Basic  Qualities  6  (Statements 
and  directives  -clear,  timely,  relevant,  complete,  and 
verified)  and  8  (Decisions  and  actions  communicated  and 
acknowledged)  could  be  combined,  but  others  disagreed.] 

I  think  that  Basic  Qualities  6  and  8  are  two  different 
things.  If  you  look  at  the  acceptable  ratings  on  the  two 
you  can  see  the  difference.  One  is  communicating  a 
decision  and  telling  the  crew  what  the  decision  is  and 
(time  allowed)  why  the  decision  was  made.  The  other  is 
the  crew  as  a  whole  discussing  the  situation  and  the  best 
or  different  course  of  action. 

There  is  apparent  overlap  among  several  Basic  Qualities 
from  an  evaluator's  perspective.  I've  given  this  a  lot 
of  thought.  But  frankly,  I  think  they  should  all  remain. 
For  instance,  I  thought  information  sought  and 
information  offered  could  be  combined,  but  after  actually 
using  them,  I  don't  think  it's  a  good  idea  to  do  any 
combining.  They  should  all  remain. 

No,  I  used  them  with  a  crew  outside  the  project.  I'm 
convinced  that  they  all  represent  different  qualities  of 
crew  coordination  and  are  all  different. 

It  was  difficult  to  apply  Basic  Quality  3,  Selection  of 
appropriate  decision  making  techniques.  It  requires 
probing  during  the  after-action  review  since  you  can't 
look  into  the  crew  member's  mind  during  the  mission. 

Need  to  emphasize  use  of  the  AAR  to  evaluate  this  Basic 
Quality. 

I'm  not  sure  that  decision  making  techniques  can  be  a 
Basic  Quality.  This  is  a  personal  technique  that  is 
accomplished  instinctively,  and  I'm  not  sure  that  it  can 
be  taught.  However,  I  still  think  it  should  be  discussed 
in  the  classroom. 

Conclusions : 

Reviewed  Basic  Qualities  6  (Statements  and  directives 
clear,  timely,  relevant,  complete,  and  verified)  and  8 
(Decisions  and  actions  communicated  and  acknowledged)  to 
ensure  that  they  were  sufficiently  distinct  from  one 
another . 

Revised  the  short  title  for  Basic  Quality  6  from  (Pos 
Comm)  to  (Info  Xfer)  to  help  clearly  distinguish  it  from 
Basic  Quality  8  (Comm  Ack) . 
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Retained  the  full  sef  of  thirteen  crew  coordination  Basic 
Qualities . 

a.  If  two  or  more  qualities  are  proposed  for  combining,  give 
specific  exairples  of  how  they  overlap.  [Remember:  Basic 
Qualities  do  not  have  to  be  con5)letely  different  from  one 
another . ] 

Specific  Comments: 

Consider  combining  Basic  Qualities  6  (Statements  and 
directives  clear,  timely,  relevant,  complete,  and 
verified)  and  8  (Decisions  and  actions  communicated  and 
acknowledged) . 

See  additional  comments  at  item  1  above. 

Conclusion:  Differentiated  the  short  titles  for  Basic 

Qualities  6  and  8. 

b.  If  a  Basic  Quality  is  proposed  for  elimination,  give 
specific  reasons  why  this  should  not  be  evaluated  as  part 
of  crew  coordination  performance.  Also  state  where  this 
aspect  of  performance  would  be  evaluated,  if  not  under 
crew  coordination. 

Summary:  Evaluators  identified  Basic  Quality  3  (  Selection 
of  appropriate  decision  making  techniques)  as  a  candidate  for 
elimination . 

Specific  Comments: 

The  decision  technique  Basic  Quality  was  difficult  for 
me.  In  most  instances,  on  the  ground  decision  making  is 
analytical,  and  in  the  air  decisions  are  made 
automatically. 

It's  a  good  academic  part,  but  it's  not  a  BQ.  All  of  the 
crews  did  this  pretty  well.  It's  a  matter  of  survival  in 
this  job. 

Decision  making  is  hard  to  see  and  evaluate.  The 
technique  I  used  was  to  focus  on  the  end  result.  I 
basically  rated  the  decision  results  but  I  asked  the 
crews  why  they  decided  things  and  how  they  came  to  a 
decision. 

Conclusion:  Reviewed  Basic  Quality  3  (  Selection  of 
appropriate  decision  making  techniques)  and  retained  it 
as  a  separate  crew  coordination  Basic  Quality. 
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2.  How  were  the  behavioral  anchors  (text  descriptions)  useful  or 
not  useful  to  you  in  achieving  reliable  and  objective  ratings 
of  crew  performance?  Please  provide  specific  exait^les,  if 
possible. 

Summary:  General  agreement  in  usefulness  of  behavioral 
anchors . 

Specific  Comments : 

I  used  the  behavioral  anchors  more  than  I  used  anything 
else.  I  considered  all  three  performance  levels  to  arrive 
at  a  rating. 

At  one  point  I  caught  myself  comparing  one  crew's 
performance  to  the  performance  of  another  crew,  and  I  had 
to  stop  that.  I  went  back  to  the  anchors  and  reviewed 
them  again.  That  helped. 

Absolute  must.  Couldn't  have  evaluated  without  them. 

I  was  concerned  that  evaluations  would  be  too  subjective, 
but  the  anchors  helped  a  lot  and  gave  me  confidence. 

Very  helpful  in  establishing  a  baseline  and  then 
adjusting  to  a  final  rating. 

The  anchors  gave  me. illustrations  and  examples  that 
helped  me  establish  a  method  for  grading.  For  example, 
pre-mission  planning  and  rehearsal  and  crew  climate. 
Useful  in  communicating  what  is  acceptable  behavior  and 
comparing  poor  versus  good  performance. 

Reviewed  behavioral  anchors  to  determine  final  grades 
because  more  time  was  available  and  behavioral  anchors 
had  more  information  upon  which  to  base  a  grade. 

The  hard  part  is  tying  the  BQs  to  the  task  specific 
behavior  in  the  ATMs. 

Conclusion:  Retained  behavioral  anchors  as  the  principal 
criteria  for  evaluating  crew  coordination  performance. 


3.  Were  some  of  the  behavioral  anchors  more  useful  than  others? 

If  so,  which  ones  were  found  to  be  more  useful?  Which  ones 
were  found  to  be  less  useful?  Please  provide  specific 
examples  of  how  you  attenpted  to  use  the  behavioral  anchors . 

Summary:  General  agreement  that  all  of  the  behavioral 
anchors  were  equally  useful. 

Specific  Comments : 

I  believe  that  I  used  them  all — some  more  than  others, 
depending  on  differences  in  each  crew's  performance. 

No.  All  were  very  useful  in  comparing  a  crew's 
performance  against  a  written  standard  instead  of  against 
the  performance  of  another  crew. 

All  were  useful  to  establish  a  baseline  for  comparing 
crew  performance. 
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Pre-mission  planning  and  rehearsal  and  after-action 
review  behavioral  anchors  were  especially  helpful. 

The  approach  I  generally  used  was  to  develop  a  sense  for 
which  Basic  Quality  or  qualities  applied;  develop  a  sense 
for  the  closest  behavioral  anchor  (e.g.,  1,  4  or  1) ;  then 
read  the  closest  anchor  description  to  select  the  anchor 
or  interpolate  to  determine  the  rating. 

Helped  separate  Basic  Qualities  initially. 

I  developed  a  tendency  to  use  a  small  set  but  not  the 
same  set  of  behavioral  anchors  routinely. 

Conclusions :  Emphasized  the  use  of  all  behavioral  anchors  in 

the  Trainer  Course  evaluation  workshop  exercises. 


4 .  How  were  the  behavioral  evaluation  factors  (bulletized 
descriptions)  useful  or  not  useful  to  you  in  achieving 
reliable  and  objective  ratings  of  crew  performance?  Please 
provide  specific  examples,  if  possible. 

Summary:  Mixed  opinion. 

Specific  Comments : 

They  were  good  but  I  used  the  behavioral  anchors.  The 
problem  with  the  bulletized  descriptions  was  that  one 
word  changed  the  factor  from  a  Superior  to  Very  Poor. 

The  rating  factors  alone  weren't  enough.  The  bulletized 
descriptions  seemed  too  subjective  and  didn't  capture 
what  was  going  on  well. 

Not  used  due  to  the  fragmented  nature  of  the 
descriptions. 

Presentation  in  class  over-emphasized  the  rating  factors. 
Should  have  spent  more  time  on  behavioral  anchors. 
Consider  using  a  structured  method  like  the  factors 
averaging  technique  to  determine  the  overall  mission 
grade  (S+,  S,  S-,  or  U) . 

Useful.  The  rating  factors  gave  me  a  practical 
comparison  for  Basic  Quality  evaluations. 

Gave  me  a  start  that  was  easy  to  interpret  and  explain  to 
the  crews . 

Used  throughout  the  missions  as  an  abbreviated  form  of 
the  behavioral  anchors.  For  example.  Crew  Preparation 
and  Composure  and  Resource  Management  factors  for  Basic 
Quality  5  (Management  of  unexpected  events) . 

Conclusion:  Retained  the  behavioral  evaluation  (rating) 

factors  as  a  screening  technique  for  identifying  the 
appropriate  Basic  Qualities  and  behavioral  anchors  to 
evaluate  crew  coordination  performance. 
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5 .  Were  some  of  the  behavioral  rating  factors  more  useful  than 
others?  If  so,  which  ones  were  found  to  be  more  useful? 

Which  ones  were  found  to  be  less  useful?  Please  provide 
specific  exairples  of  how  you  attenpted  to  use  the  behavioral 
rating  factors . 

Summary:  Mixed  opinion. 

Specific  Comments: 

I  believe  at  one  time  or  another  they  were  all  very 
important.  They  gave  me  a  start  point  to  determine  an 
evaluation  rating. 

I  don't  think  that  there  were  any  rating  factors  that 
were  more  useful  than  any  others. 

Task  prioritization  and  awareness  level  of  crew  were  the 
two  most  useful  rating  factors. 

Factors  were  good  for  initial  use  in  the  simulator  until 
I  was  fully  familiar  with  the  behavioral  anchors. 

I  used  the  rating  factors  to  observe  the  crew's  actions 
and  relate  them  to  the  Basic  Qualities;  then  I  referred 
to  the  behavioral  anchors  to  determine  the  final  grades. 

Conclusion:  Emphasized  the  use  of  rating  factors  as  a 

screening  technique  to  apply  behavioral  anchors  in  the 
Trainer  Course  evaluation  workshop  exercises. 


6 .  How  did  you  use  the  evaluation  factors?  Did  you  use  them  to 
instruct  in  the  classroom  or  simulator?  Crew  debriefing? 

Summary:  General  agreement  on  having  used  the  evaluation 

(rating)  factors  for  classroom  instruction  with  some 
references  made  during  crew  debriefings. 

Specific  Comments: 

The  evaluation  (rating)  factors  were  very  good  to  use  as 
classroom  instruction  material.  They  were  presented  in 
formal  training  aids  to  link  instructional  topics  to  the 
Basic  Qualities. 

Instructing  in  the  course.  Taught  as  '-'effectiveness 
factors" . 

I  used  them  occasionally  to  structure  the  crew  debriefing 
and  referenced  them  when  discussing  the  mission  with  the 
crews . 

Suggest  that  some  of  the  rating  factors  could  be  used  to 
develop  an  After  Action  Review  checklist. 

I  used  the  Basic  Quality  definitions  in  the  simulator  and 
crew  debriefing. 

Conclusion:  Retained  evaluation  (rating)  factors  in  the 
classroom  instruction  as  abbreviated  forms  of  superior, 
acceptable,  and  very  poor  behavioral  anchor  descriptions. 
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7.  Should  we  continue  to  have  both  the  (rating)  factors  and 

(behavioral)  anchors  for  the  evaluation  guidance?  If  we  use 
only  one,  which  one  would  you  choose? 

Summary:  General  agreement  on  behavioral  anchors  as  the 
criteria  for  evaluating  crew  coordination  performance. 

Specific  Comments : 

I  was  confused  by  the  fact  that  there  were  two  evaluation 
methods;  specifically,  the  rating  factors  and  the 
behavioral  anchors.  I  wasn't  sure  at  first  and  used 
both. 

The  rating  factors  were  very  specific  and  the  anchors 
gave  general  guidelines.  I  liked  the  behavioral  anchors 
and  think  they  are  appropriate. 

I  stopped  using  the  rating  factors  and  used  only  the 
behavioral  anchors.  I  think  the  behavioral  anchors 
allowed  me  to  give  fairer  ratings. 

I  used  both,  but  if  I  had  to  choose,  I  would  choose  the 
behavioral  anchors.  I  thought  they  were  more  practical 
in  evaluating  crew  performance. 

More  comfortable  with  the  paragraph  format  (behavioral 
anchors)  versus  the  bullets  (rating  factors) .  There  is  a 
danger  of  focusing  on  an  isolated  fact  or  word  in  the 
bulletized  format. 

Perhaps  it  would  be  good  to  have  them  both  and  let  the 
more  experienced  raters  use  only  the  behavioral  anchors. 
Use  only  the  behavioral  anchors.  The  rating  factors  are 
only  needed  in  the  classroom  instruction 

If  forced  to  select  only  one  method — keep  the  behavioral 
anchors  because  with  experience  they  provide  more 
information. 

These  evaluations  have  a  subjective  part  to  them.  The 
evaluation  factors  are  too  restrictive,  a  black  and  white 
decision.  The  rating  factors  didn't  work  for  me. 

Conclusion:  Emphasized  behavioral  rating  anchors  as  the 

principal  criteria  for  evaluating  crew  coordination 
performance . 


8 .  Do  the  audio  segments  [used  in  the  Instructor  Course 
evaluation  workshop]  provide  adequate  opportunity  for 
practicing  your  application  of  the  rating  scales? 

Summary:  Mixed  opinion. 

Specific  Comments : 

I  think  it  would  be  better  to  use  only  text  descriptions 
because  we  all  have  stereotypes  in  mind  that  effect  our 
ratings  when  it  comes  to  videotapes. 

The  script  was  good  to  use  with  the  audio  tapes. 

Yes,  but  it  would  have  been  better  to  have  more  examples. 
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Yes,  they  were  better  than  written  examples  but  video 
examples  would  be  best. 

Entertaining,  but  unable  to  grade  performance  without 
seeing  crew  actions. 

Once  I  got  into  the  behavioral  anchors,  then  I  really 
understood  how  to  evaluate. 

No,  applying  the  rating  scales  only  became  clear  once  we 
got  in  the  simulator. 

Need  video  segments  for  effective  practice  exercises. 
Conclusions : 

Identified  candidate  video  segments  from  the  crew 
coordination  testbed  missions  for  use  in  the  Trainer 
Course  evaluator  workshop  exercises. 

Included  narrative  scripts  from  the  candidate  video 
segments  in  the  evaluator  workshop  exercises. 

Included  an  evaluator  practice  session  in  the  simulator 
as  part  of  the  Trainer  Course. 

a.  Were  the  13  Basic  Qualities  addressed  in  an  adequate 
fashion?  If  not,  how  could  the  segments  be  expanded  to 
address  each  Basic  Quality? 

Summary:  Mixed  opinion. 

Specific  Comments: 

Yes,  I  think  that  there  was  enough  on  each  segment. 
However,  if  it's  possible  to  add  more  video  examples  to 
explain  how  to  evaluate  the  BQ's,  they  should  be  added. 
Not  really,  need  more  segments  or  longer  segments. 

Basic  Qualities  were  covered  very  well. 

Conclusion:  Identified  candidate  video  segments  from  the 

crew  coordination  testbed  missions  for  use  in  the  Trainer 
Course  evaluator  workshop  exercises. 

b.  Would  you  find  videotapes  to  be  more  useful?  If  so,  what 
type  of  vignettes  would  you  recommend  be  included? 

Summary:  General  agreement  that  videotapes  would  be  superior 

to  audio  tapes. 

Specific  Comments : 

Use  a  video  segment  to  walk  through  the  entire  evaluation 
process,  step-by-step. 

Yes,  I  would  like  to  have  videotapes  for  the  evaluation 
exercises.  Suggest  adding  contrasting  excerpts  from  the 
before  and  after  training  flights  of  selected  testbed 
crews . 

Suggest  a  "branching"  video  that  includes  both  good  and 
bad  crew  performance  in  a  series  of  related  situations. 
More  video  segments  of  accidents  or  incidents. 
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Consider  developing  videos  that  show  an  event  with 
flashbacks  to  premission  planning  or  other  crew 
coordination  activities  that  would  have  assisted  in 
resolving  the  situation. 

Video  would  be  fine  but  audio  provides  the  same  points. 
Audience  concentrates  more  on  performance  when  audio  is 
the  primary  media. 

Consider  developing  a  videotape  that  has  two  female-rated 
aviators  and  see  how  the  crew  is  rated  by  male  and  female 
evaluators . 

Conclusions : 

Identified  candidate  video  segments  from  the  crew 
coordination  testbed  missions  for  use  in  the  Trainer 
Course  evaluator  workshop  exercises. 

Included  narrative  scripts  from  the  candidate  video 
segments  in  the  evaluator  workshop  exercises. 

Identified  candidate  video  segments  for  possible  use  in 
developing  a  "branching"  videotape  for  training  and/or 
evaluation  exercises. 


9.  Was  the  7-point  scale  a  good  choice  for  rating  crew 

coordination  Basic  Qualities?  Were  the  descriptors  (very 
poor,  poor,  marginal,  adequate,  acceptable,  good,  very  good, 
superior)  for  each  number  helpful?  Do  you  have  any 
suggestions  for  in^srovement? 

Summary:  Strong  agreement  that  the  scale  and  anchor 

descriptions  provide  the  basis  for  objective,  reliable 
ratings . 

Specific  Comments: 

Excellent.  Don't  change  a  thing. 

There  is  enough  difference  between  7,  4,  and  1  to  decide 
where  to  interpolate. 

Had  some  initial  problems  interpreting  between  1  to  4  and 
4  to  7  but  with  practice,  it  was  OK. 

The  scale  range  is  good. 

Consider  providing  the  crews  with  the  behavioral  anchor 
descriptions  not  just  the  abbreviated  rating  factors. 

See  related  comments  at  item  IV,  10a. 

Conclusions : 

Retained  the  7-point  rating  scale  and  behavioral  anchor 
descriptions  for  evaluating  initial,  refresher,  and 
continuation  crew  coordination  training. 

Included  the  behavioral  anchor  scale  and  descriptions  in 
the  Student  Course  materials. 
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10.  Think  back  to  each  of  the  13  Basic  Qualities.  Each  has  a 
behavioral  anchor  for  numbers  1 ,  4 ,  and  7 . 

a.  Were  the  behavioral  anchors  helpful? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

Yes.  See  related  comments  at  item  9. 

They  gave  me  the  descriptions  and  examples  I  needed  to 
rate  crew  coordination  skills. 

Too  much  information  to  use  in  the  cockpit  but  great  for 
determining  final  grades. 

Best  source  to  evaluate  pre-mission  planning  and 
rehearsal  and  after  action  review  activities. 

See  related  comments  at  items  2  and  3. 

Conclusion:  Retained  behavioral  anchor  descriptions  with 
minor  changes. 

b.  What  are  your  suggestions  for  improving  them  (be  as 
specific  as  possible  so  we  can  incorporate  your 
suggestions) ? 

Summary:  General  agreement  that  few  if  any  changes  are 

needed. 

Specific  Comments : 

Consider  adding  checklists  (e.g.,  hover  power  check, 
landing  check,  etc) . 

I  would  have  liked  to  use  a  plus  or  minus  in  conjunction 
with  the  scale  rating  (e.g.,  3+,  4-,  etc).  I  recognize 
that  this  would  create  a  21  point  scale  but  the  benefit 
might  be  worth  it.  For  example,  if  a  crew  had  difficulty 
with  an  emergency  procedure  and  didn't  do  the  procedure 
correctly,  or  didn't  do  it  at  all,  but  very  good  crew 
coordination  was  the  only  thing  that  enabled  that  crew  to 
make  a  safe  landing,  the  rating  for  them  under  BQ  4; 
management  of  unexpected  events  might  be  a  3+. 

Consider  developing  descriptions  for  intermediate  ratings 
(i .e. ,  2,  3,  5,  &  6)  . 

I  think  they  were  very  clear  and  easy  to  use.  No 
suggestions  for  improvement. 

Conclusions : 

Retained  behavioral  anchor  descriptions  with  minor 
changes . 

Included  the  use  of  checklists  in  Basic  Quality  10, 
Crewmember  actions  mutually  cross-monitored. 
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11.  Think  back  to  each  of  the  13  Basic  Qualities.  Each  has 
evaluation  (rating)  factors  for  numbers  1,  4,  and  7. 

a.  Were  the  evaluation  (rating)  factors  helpful? 

Summary:  Mixed  opinion. 

Specific  Comments : 

Yes,  at  first  they  were  real  helpful  and  I  needed  them 
but  later  I  didn't  use  them. 

At  first  I  used  them  to  compare  performance  and  derive  a 
guideline  for  evaluations.  However,  I  don't  think  they 
were  as  specific  as  the  behavioral  anchors.  I  preferred 
the  behavioral  anchors. 

Served  as  memory  joggers  and  a  shorthand  checklist  of 
behavioral  anchors. 

No,  did  not  use  them  for  evaluations. 

Useful  in  teaching  the  course.  Explanations  are  short, 
easy  to  read  outlines  of  performance. 

See  related  comments  at  items  4,  5,  and  6. 

Conclusions : 

Emphasized  behavioral  anchors  as  the  principal  criteria 
for  evaluating  crew  coordination  performance. 

Retained  evaluation  (rating)  factors  for  the  classroom 
instruction. 

b.  What  are  your  suggestions  for  iitproving  them  (be  as 
specific  as  possible) ? 

Summary:  Mixed  opinion. ■ 

Specific  Comments : 

Include  the  rating  factors  in  the  behavioral  anchor 
descriptions . 

Change  the  name  from  rating  factors  to  "effectiveness 
factors . " 

No  suggestions  for  improvement. 

Conclusion:  Retained  in  classroom  instruction  without 
modification. 


12 .  Were  you  reluctant  to  give  crews  ratings  below  "fully 
acceptable"?  If  yes,  why? 

Summary:  No.  Strong  agreement. 

Specific  Comments : 

No.  I  gave  low  ratings  when  I  saw  them. 

I  had  no  hesitation  to  give  low  ratings  where  appropriate 
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Yes,  initially  I  was  somewhat  reluctant  given  the  S  or  U 
grading  mindset  because  a  "U"  means  that  a  crew  goes  to 
RL  3  and  impacts  the  entire  unit. 

Conclusion:  Retained  the  behavioral  anchor  descriptions  and 

7-point  rating  scale  for  evaluating  initial,  refresher,  and 
continuation  crew  coordination  training. 


13 .  If  you  had  experience  using  both  the  behavioral  anchors  and 
the  evaluation  factors,  do  you  think  that  one  or  the  other 
methods  influenced  you  to  give  higher  or  lower  grades?  For 
example,  if  you  used  the  evaluation  factors,  do  you  think 
that  led  you  to  give  higher  or  lower  grades  than  if  you  had 
used  the  behavioral  anchors? 

Summary:  General  agreement  that  the  behavioral  anchors 
produce  more  reliable  evaluations  of  crew  coordination 
performance . 

Specific  Comments : 

Yes,  there  are  differences.  The  rating  factors  are  very 
specific.  They  either  fall  into  a  factor  or  not.  The 
behavioral  anchors  gave  me  wider  latitude  and  allowed  me 
to  compare  performance  in  relation  to  the  anchored 
standard.  I  could  easily  decide  if  crews  were  close  to  a 
description  and  then  decide  if  they  were  a  little  better 
or  worse  than  the  behavioral  anchor. 

The  description  of  the  behavioral  anchors  had  more 
underlying  meaning  and  I  found  them  to  be  more  effective 
than  the  rating  factors. 

I  moved  away  from  using  the  rating  factors  and  used  the 
behavioral  anchors  instead.  Had  I  continued  to  use  the 
rating  factors,  I  probably  would  have  been  more  inclined 
to  give  fours.  With  the  behavioral  anchors  I  was  more 
able  to  select  a  3  or  a  5. 

Wording  of  the  behavioral  anchors  provided  sufficient 
range . 

Rating  factors  and  behavioral  anchors  seem  to  complement 
rather  than  conflict  or  contradict.  No  affect  on  grades 
given. 

I  believe  the  behavioral  anchors  let  me  give  a  more 
precise  evaluation  because  of  the  description  and 
examples  for  each  rating  level. 

The  rating  factors  led  me  to  give  a  higher  grade  because 
they  were  not  as  specific  as  the  anchors. 

N/A  since  didn't  use  the  behavioral  factors. 

I  think  the  behavioral  anchors  would  lead  to  lower 
ratings  because  they  give  a  more  complete  picture  of  what 
evaluators  see. 
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Conclusion:  Retained  the  behavioral  anchor  descriptions  and' 

7-point  rating  scale  for  evaluating  initial,  refresher,  and 
continuation  crew  coordination  training. 


14.  How  often  did  you  refer  to  the  explanations  in  the  behavioral 
anchors? 

Summary:  General  agreement  on  frequent  use  (every 
evaluation)  initially  with  continued  referencing  of  them  by 
experienced  evaluators. 

Specific  Comments : 

All  the  time.  Some  Basic  Qualities  more  than  others. 
Frequently  at  first,  then  only  by  exception. 

Very  often  at  first,  then  infrequently  during  last  two 
evaluations . 

Basically  everyday,  if  just  to  reinforce  the  description 
in  my  own  mind.  I  did  this  to  be  sure  I'd  give  the  crew 
the  fairest  grade. 

Almost  every  training  ride  in  the  simulator. 

For  the  initial  evaluations,  I  used  all  of  the  BQ 
descriptions  all  of  the  time.  Then  some  things  sunk  in 
and  I  didn't  have  to  refer  to  them  as  often. 

I  reviewed  them  initially;  looked  at  them  again  for  the 
first  ride;  and  then  did  not  have  to  refer  to  them  again. 
Almost  always.  I  would  refer  to  the  rating  factors 
first,  then  I  would  refer  to  the  behavioral  anchors 
unless  it  was  clearly  covered  in  the  rating  factors. 

Conclusion :  Emphasized  behavioral  rating  anchors  as  the 

principal  criteria  for  evaluating  crew  coordination 
performance . 


15.  How  often  did  you  refer  to  the  evaluation  (rating)  factors? 

Summary:  General  agreement  on  occasional  use  (very  little) 
at  first  and  then  dropped  during  evaluations.  However,  they 
are  useful  for  classroom  instruction. 

Specific  Comments : 

Whenever  I  needed  to  grade  a  task  for  the  first  time. 

At  first  I  used  them,  then  I  dropped  them. 

Initially,  I  referred  to  both  evaluation  factors  and 
behavioral  anchors;  but  as  time  went  by  and  I  became  more 
comfortable  and  knowledgeable  I  referred  to  the 
evaluation  factors  less  and  less. 

Maybe  once. 

Never. 

Only  in  class. 
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Conclusion:  Retained  evaluation  (rating)  factors  in  the 

classroom  instruction. 


16.  Did  the  frequency  of  referral  to  the  behavioral  anchor 

descriptions  change  over  time  (e.g.,  less  referral  to  the 
anchor  descriptions  with  experience) ? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

See  related  comments  at  item  14. 

Yes,  with  more  experience  the  less  I  referred  to  them. 
Yes,  especially  for  those  that  I  used  the  most. 

Conclusions : 

Emphasized  behavioral  anchors  as  the  principal  criteria 
for  evaluating  crew  coordination  performance. 

Encouraged  instructors  to  refer  to  the  behavioral  anchor 
descriptions  frequently  to  avoid  the  habit  of  comparing 
crews  to  one  another  instead  of  comparing  performance  to 
the  behavioral  anchor  description. 


17.  Did  the  frequency  of  referral  to  the  evaluation  (rating) 

factors  change  over  time  (e.g.,  less  referral  to  the  factor 
descriptions  with  experience) ? 

Summary:  Mixed  opinion. 

Specific  Comments : 

See  related  comments  at  item  15. 

Yes,  maybe  a  little. 

No. 

Not  applicable.  Did  not  use  rating  factors  other  than  in 
the  classroom. 

Conclusion:  Retained  evaluation  (rating)  factors  in  the 
classroom  instruction. 


18 .  Are  there  any  aspects  of  crew  performance  not  adequately 
covered  in  the  13  Basic  Qualities? 

Summary:  No.  General  agreement  on  emphasizing  the  crew 
coordination  aspect  of  after  action  review  and  including  the 
use  of  checklists  in  existing  Basic  Quality  descriptions. 

Specific  Comments : 

No,  full  coverage. 

The  13  adequately  cover  all  aspects. 
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Relook  the  wording  and  substance  of  Basic  Quality  13, 

Crew  level  AARs  accomplished,  to  consider  both  crew 
coordination  actions  and  flight  tactics  and  techniques 
for  each  mission  segment. 

How  about  the  use  of  checklists  and  procedures?  There 
was  nothing  in  here  that  allowed  me  to  remark  about  "this 
is  something  that  every  aviator  ought  to  know  coming  out 
of  flight  school." 

Seemed  to  cover  everything;  may  get  interesting  when 
nonrated  crewmembers  are  included  in  the  program. 

'Conclusions : 

Developed  and  included  an  AAR  checklist  and  discussion  on 
its  use  in  the  course  of  instruction. 

Included  the  use  of  checklists  in  Basic  Quality  10, 
Crewmember  actions  mutually  cross-monitored. 

a.  If  so,  would  you  address  these  aspects  within  the  context 
of  one  of  the  existing  Basic  Qualities?  Please  be 
specific. 

Summary:  Does  not  apply. 

b.  If  so,  would  you  address  these  aspects  as  a  separate 
Basic  Quality?  Please  be  specific. 

Summary:  Does  not  apply. 
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V.  Modified  Grade  Slips 

1 .  Was  the  format  of  the  grade  slip  understandable  and  easily 
used?  Any  specific  suggestions  for  in^srovement? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

Yes,  easily  understandable.  It's  really  not  much 
different  from  the  current  grade  slips. 

Easy  to  use.  Like  using  the  maneuver  procedure  grade 
slip . 

Yes,  I  thought  it  was  easy  to  understand  and  very  similar 
to  the  one  we  use  now. 

Easy  to  use.  It  followed  the  already  familiar  maneuver 
procedure  grade  slip. 

Easy  and  understandable.  No  problems. 

Basic  Qualities  list  at  the  bottom  of  the  grade  slip  is 
sufficient  for  in  flight  reference.  I  referred  to  the 
full  descriptions  before  signing  the  grade  slip. 

Consider  shading  the  scenario  driven  tasks.  [This  was 
done  for  the  testbed  pre-  and  post-training  evaluations.] 
Consider  including  the  Basic  Qualities  reference  line  in 
the  ATM  maneuver  procedure  grade  slip. 

Conclusion :  Retained  the  Aircrew  Coordination  Training 

Grade  Slip  for  evaluating  initial  and  refresher  crew 
coordination  training. 


2.  Did  you  weight  flying  skills  and  aircrew  coordination  skills 
differently?  Did  you  give  them  equal  weight? 

Summary:  Mixed  opinion. 

Specific  Comments: 

I  tried  to  separate  them,  but  sometimes  they  are  very 
bound  up.  For  instance,  on  approach  if  the  pilot  flying 
was  too  fast,  he  had  to  use  his  skills  to  slow.  But  the 
pilot  not  flying  should  have  announced  the  landing 
earlier  so  the  aircraft  wasn't  going  so  fast  in  the  first 
place--that ' s  the  crew  coordination  piece. 

It  varied.  For  the  testbed,  I  weighed  crew  coordination 
more,  but  for  some  specific  maneuvers,  I  weighed  them 
about  50-50,  and  for  other  things  'I  leaned  toward  the 
flight  skill.  I  always  considered  crew  coordination. 
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Weighing  shifted  during  the  course.  I  made  a  conscious 
decision  to  weight  crew  coordination  heavily  later  in  the 
course,  e.g.. 


prepost 

flying  skills  80%  20% 

crew  coordination  20%  80% 


Equal  weight  e.g.,  non-precision  approach,  flying  skills 
on  breaking  altitude,  crew  coordination  on  cross  monitor 
performance . 

For  the  purpose  of  the  testbed,  I  weighed  aircrew 
coordination  heavier  than  flying  skills.  However,  for 
some  specific  maneuvers  (i.e.,  major  malfunction, 
inadvertent  IMG,  radio  navigation,  nonprecision  approach) 
I  weighted  flying  skills  just  as  heavy  if  not  heavier 
than  crew  coordination. 

I  concentrated  mainly  on  crew  coordination  and  rarely 
commented  on  flight  skills. 

Yes,  I  weighted  them  differently.  The  course  purpose  was 
to  look  at  crew  coordination.  I  gave  more  weight  to  the 
crew  coordination  aspects. 

Conclusion :  Emphasized  the  Aircrew  Training  Manual 

evaluation  principle  that  the  evaluator's  subjective  analysis 
and  judgment  are  central  to  reliable  evaluations  of  crew 
performance . 

3 .  Was  the  satisfactory  plus  (S+) ,  satisfactory  (S) ,  and 

satisfactory  minus  (S-)  grading  system  helpful?  Would  you 
like  to  use  S+,  S,  and  S-  for  APART  rides? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

I  think  it  was  helpful,  especially  when  using  the  two  BQs 
as  the  basis  for  the  grade.  If  we  only  have  an  S,  it's 
not  enough  information. 

I  would  like  the  S-  instead  of  the  U  available.  It  seems 
fairer.  A  crew  may  be  weak  and  while  I  don't  want  to 
give  them  a  U,  an  S  is  too  good. 

For  a  teaching  tool,  the  S+  and  S-  are  good  to  show 
progression,  but  for  an  evaluation  tool,  I'm  not  sure. 

I  thought  it  was  very  helpful  during  these  evaluations. 

I  would  like  to  see  it  used  in  the  field  with  guidelines 
that  an  S-  has  to  be  corrected  within  a  certain  amount  of 
time.  Also,  if  we  used  S+,  it  would  allow  us  to  track 
the  strong  aviators  and  maybe  spread  the  talent. 

The  S-  would  help  me  "go  home  at  night."  For  example,  I 
would  like  to  have  the  option  to  give  a  S-  instead  of  an 
S  when  it  takes  a  crew  three  attempts  to  perform  a 
maneuver  to  standard. 
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Very  helpful  for  evaluation;  can  give  a  U  for  a  maneuver 
without  grading  the  overall  mission  Unsatisfactory. 

Not  suitable  for  individual  APART  given  the  current 
guidance . 

The  expanded  grading  system  gives  IPs  leeway  to  identify 
strengths  and  weaknesses  without  issuing  Unsatisfactory 
grades . 

I  would  love  to  use  this  system  on  APART.  Some  aviators 
are  satisfied  to  be  within  the  limits  while  others  strive 
to  be  exact. 

Yes,  it  was  very  helpful.  I  would  like  to  see  this  used 
in  the  field  with  a  guideline  of  options  to  be  followed 
should  someone  receive  an  S-.  For  example,  if  an  S-  is 
received  for  an  oral  evaluation,  that  person  must  be 
reevaluated  on  the  oral  within  ten  days  from  the  date  of 
the  S-.  If  the  next  evaluation  is  no  better,  then  the 
grade  automatically  becomes  a  U. 

Being  able  to  use  an  S+  would  provide  a  way  to  identify 
and  track  strong  performing  crews. 

Yes,  it  is  more  specific  and  gives  something  the  shoot 
for  (S+)  . 

Recommend  changing  to  the  expanded  grading  system,  but  it 
would  require  a  lot  of  changed  thinking.  Probably  best 
to  use  it  only  for  crew  coordination  training. 

Conclusions : 

Retained  the  expanded  grading  system  (S+,  S,  S-,  U)  for 
evaluating  initial  and  refresher  crew  coordination 
training. 

Recommended  that  the  USAAVNC  revise  the  APART  field 
grading  system  to  allow  for  use  of  the  expanded  grading 
system  (S+,  S,  S-,  U) . 


4.  Would  you  have  liked  to  use  the  Basic  Quality  notations  (1,2, 

.  .  .13)  for  both  positive  and  negative  crew  coordination 
behaviors?  If  yes,  would  using  a  "+"  or  sign  next  to  each 

BQ  associated  with  ATM  Task  performance  be  a  reasonable 
marking  technique,  or  do  you  think  it  would  be  too  corrplex? 

Summary:  Mixed  opinion. 

Specific  comments. 

Yes,  I  think  that  would  be  a  good  idea.  I  don't  think  it 
would  be  too  complex.  We  could  do  that.  For  now  I  used 
the  worksheet  and  the  comments . 

The  technique  would  not  be  too  complex.  It  would  become 
easier  over  time  and  could  provide  better  information  to 
make  final  grades  and  ratings. 
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Yes,  I  think  it's  a  good  idea.  I  think  it  provides  a 
better  measure  of  crew  coordination  abilities.  It  is 
just  as  important  to  note  positive  as  well  as  negative 
performance . 

Not  necessary  since  a  grade  with  a  +  or  -  indicates  the 
weight  and/or  direction  of  the  contributing  Basic 
Qualities . 

No,  I  wrote  up  exceptional  performance,  both  superior  and 
poor,  on  the  comment  sheet. 

I  think  that  the  number  ratings  on  the  grade  slip  should 
be  sufficient  for  evaluating  Basic  Qualities,  both 
positive  and  negative. 

Conclusion:  Included  an  option  for  evaluators  to  use  a  "+" 

or  sign  next  to  the  Basic  Quality  numbers  (for  example,  S 

+7,  -10)  to  identify  both  positive  and  negative  crew 
coordination  contributions  to  task  performance. 


5.  When  yon  gave  an  overall  mission  grade,  what  were  your 
criteria? 

Summary:  General  agreement  that  criteria  for  overall  mission 
grade  is  a  synthesis  of  the  ATM  standards  for  the  tasks 
performed  supported  by  Aircrew  Coordination  Training  Grade 
Slips  and  evaluator  worksheets. 

Specific  Comments: 

It  was  a  combination  of  everything  including  crew 
coordination,  flight  skills,  mission  success.  I  weighed 
them  about  equally. 

Worksheet  notes  to  determine  Basic  Quality  ratings  (i.e. 
the  number  of  +  and/or  -  from  task  grades) . 

Averaged  Basic  Quality  ratings  to  determine  overall  grade 
(e.g. ,  4  =  S)  . 

I  compared  the  ATM  crew  coordination  description  and  the 
standards  for  the  maneuver  with  what  I  observed. 

Primarily,  I  used  the  worksheets  as  a  guide. 

I  mixed  crew  coordination  and  flying  skills  with  more 
weight  given  to  crew  coordination. 

^  -  I  used  the  Basic  Quality  ratings. 

I  considered  the  importance  of  the  task  grades. 

Became  more  influenced  by  Basic  Quality  ratings  rather 
than  the  ATM  task  grades. 

Used  grades  from  the  grade  slip,  comments  on  worksheet, 
and  ATM. 

Considered  the  overall  mission.  A  "U"  on  a  task  did  not 
negatively  affect  the  overall  grade  but  was  considered 
with  everything  else. 

Conclusion:  Retained  the  Aircrew  Coordination  Training 

Grade  Slip  and  evaluator  worksheets  for  evaluating  initial 
and  refresher  crew  coordination  training. 
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6.  Did  you  find  the  Comment  Slip  useful?  If  not,  do  you  have  any 
specific  suggestions  to  irtprove  its  use? 

Summary:  Yes.  Strong  agreement. 

Specific  Comments: 

I  used  it  everyday.  It  worked  fine. 

Included  general  comments  on  the  flight. 

Used  it  to  explain  S-,  S+,  or  U  grades  on  maneuvers. 
Captured  Basic  Quality  contributors  to  task  grades. 

Yes,  I  used  it  with  every  flight. 

Comments  are  always  useful  (for  example,  strengths  and 
weaknesses) . 

Yes,  for  annotating  deficiencies  or  exceptional 
performance . 

No  suggestions  for  improvement. 

Conclusion:  Retained  DA  Form  7121-R,  Battle-Rostered  Crew 

Evaluation  Training  Grade  Slip  for  use  in  evaluating  crew 
coordination  initial,  refresher,  and  continuation  training. 
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VI .  General  Observations 

1 .  What  is  your  overall  iit^ression  of  the  adequacy  of  the 

aircrew  coordination  training  provided?  Do  you  have  any 
recommendations  for  inprovement? 

Summary:  Strong  agreement  that  the  training  provided  is 
fully  adequate  to  implement  crew  coordination  training  in 
aviation  units. 

Specific  Comments : 

The  positive  response  from  the  crews  and  the  increase  in 
performance  speaks  for  itself. 

The  information  is  being  taken  back  to  the  flight  line 
right  now.  IPs  will  use  it  individually  between  now  and 
when  the  package  is  formally  fielded. 

I  think  this  is  a  good  program.  We  should  be  doing  it 
now.  Good  crews  have  been  doing  it  without  realizing  it. 
Now  it's  organized  and  we  can  understand  it  better. 
Effective  crews  must  practice  crew  coordination  or  they 
are  looking  for  an  accident. 

For  this  kind  of  training  and  evaluation,  I  thought  it 
worked  very  well.  I  think  it's  really  important. 

We  saw  so  much  improvement  after  only  three  days  in  a 
classroom  and  three  sessions  in  a  simulator  cockpit.  I 
think  that  speaks  volumes. 

Successfully  demonstrated  that  we  can  train  crew 
coordination  in  units. 

Great  impact  on  self  and  crews. 

IP/UT  mix  for  team  training  was  very  effective. 

Scenario  development  by  instructor  teams  was  good  to 
capture  different  ideas. 

Very  well  organized  and  thorough.  I  was  impressed  with 
the  whole  program.  It's  professionally  organized  and 
taught.  I'm  not  sure  it  could  be  done  any  better. 

This  stuff  really  works. 

Enhances  everybody's  ability  to  perform  safely. 

Good.  Consider  adding  more  practical  exercises.  [See 
related  comments  and  conclusions  at  items  III,  3  and  12 
through  15  in  the  Summary  of  Crewmember  Exit  Interview 
Results . ] 

Very  good.  Need  to  get  the  chain  of  command  involved 
from  the  start  when  a  unit  starts  the  training  program. 
[See  related  comments  and  conclusions  at  item  II,  6  in 
the  Summary  of  Crewmember  Exit  Interview  Results.] 

My  concern  is  with  the  implementation  of  the  TC's  1-210, 
1-212  upon  us,  how  do  we  go  about  bringing  the  rest  of 
our  units  up  to  this  level?  It  would  be  a  shame  to  have 
all  of  this  material  available,  and  not  use  it. 
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One  thing  I  liked  about  the  course  is  that  it  wasn't 
written  at  the  6th  grade  level.  I  think  it's  important 
to  keep  the  significant  information;  don't  just  focus  on 
the  task  details.  Aviators  should  have  both  for  complete 
understanding.  We  should  train  to  the  maximum;  not  the 
minimum. 

I  wonder  how  difficult  it's  going  to  be  to  incorporate 
crew  chiefs  into  this  training.  They  should  be  in  the 
same  class,  but  the  simulator  work  is  not  possible.  We 
will  have  to  involve  crew  chiefs  aboard  the  aircraft. 

[See  related  comments  and  conclusions  at  item  IV,  1  in 
the  Summary  of  Crewmember  Exit  Interview  Results.] 

Conclusion:  Revised  the  validation  testbed  course  of 
instruction  to  final  for  use  as  the  USAAVNC's  Exportable 
Training  Package  for  initial  and  refresher  crew  coordination 
training. 


2.  What  is  your  overall  impression  of  the  adequacy  of  the 
evaluation  training  provided?  Do  you  have  any 
recommendations  for  iit5)rovement? 

Summary:  Strong  agreement  that  the  training  provided  is 

fully  adequate  to  permit  unit  IPs  and  UTs  to  conduct 
objective,  reliable  evaluations  of  crew  coordination 
performance. 

Specific  Comments: 

It's  good.  Initially,  it  seemed  like  a  lot,  but  the  more 
we  practiced,  the  easier  it  became. 

I  was  very  comfortable  with  my  training.  It  was  just 
great.  I  could  do  everything  I  needed  to  do  and  I  could 
give  fair  grades. 

Comfortable  evaluating  crews. 

Evaluators  should  have  the  full  set  of  training  missions 
instead  of  only  the  scenario  familiarization  mission. 

[See  related  comments  and  conclusions  at  item  I,  8.] 

About  the  right  amount  of  time.  [See  related  comments 
and  conclusions  at  items  I,  1  and  9.] 

Practical  exercises  in  the  evaluator  workshop  were  good. 
Need  to  do  at  least  3  or  4  exercises. 

Need  to  emphasize  the  capability  [to  record  simulator 
time  on  evaluator  worksheets  to  locate  events  on 
videotape  for  review]  in  the  Trainer  Course  (e.g., 
evaluation  workshops) . 

Conclusions : 

Added  a  complete  scenario  familiarization  mission  to  the 
Trainer  Course  with  instructors  performing  both 
crewmember  and  evaluator  duties  critiqued  by  course 
trainers . 
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Included  all  Student  Course  training  and  evaluation 
missions  in  the  Trainer  Course. 

Revised  the  evaluation  training  sections  of  the 
validation  testbed  course  of  instruction  to  final  for  use 
as  the  USAAVNC's  Exportable  Training  Package  for  initial 
and  refresher  crew  coordination  training. 


3 .  What  is  your  overall  inpression  of  the  conduct  of  the  aircrew 
coordination  evaluations?  Do  you  have  any  recommendations 
for  irr5>rovement? 

Summary:  Strong  agreement  that  evaluations  were  objective 

and  reliable. 

Specific  Comments : 

1  thought  the  evaluations  were  effective.  They  worked 
well . 

My  overall  impression  is  that  the  evaluations  were  done 
very  properly  and  professionally.  The  only  caution  is 
that  evaluators  should  guard  against  imposing  their 
personal  crew  coordination  techniques  before  the  crews 
have  time  to  practice  using  the  Basic  Qualities  for 
themselves . 

I  think  it's  important  to  keep  the  crews  together  and 
keep  them  battle-rostered.  I  also  think  that  we  should 
not  swap  crews  during  the  training.  Perhaps  a  better  idea 
is  to  split  up  the  evaluators. 

Reliable,  objective,  and  fair. 

Evaluating  in  the  cockpit  while  on  the  controls  as  a 
crewmember  will  be  a  challenge 

The  video  segments  and  testbed  data  collection  logger 
sheets  were  a  great  help  (e.g.,  cleared  circling,  busted 
MDA,  crew  disagreements  over  task  completion) . 

At  first,  I  didn't  think  there  was  any  way  to  fairly 
evaluate  crew  coordination.  After  seeing  this  technique, 
I  am  convinced  that  it  is  fair  and  necessary. 

Evaluations  covered  all  aspects  of  crew  coordination  and 
allowed  me  to  use  my  own  judgement  as  required. 

Need  to  provide  students  with  behavioral  anchor 
descriptions . 

Conclusions : 

Included  the  behavioral  anchor  scale  and  descriptions  in 
the  Student  Course  materials. 

Retained  the  testbed  evaluation  methods  and  materials  for 
use  in  the  USAAVNC's  Exportable  Training  Package  for  crew 
coordination  initial  and  refresher  training  evaluations. 
Retained  the  testbed  evaluation  methods  and  materials  for 
use  in  the  USAAVNC's  Exportable  Evaluation  Package  for 
crew  coordination  continuation  training  evaluations. 
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4.  If  this  training  package  is  fielded  Amy-wide,  should  there 
be  a  system  to  track  testbed  participants  in  longitudinal 
studies  of  the  Amy's  crew  coordination  program? 

Summary:  Yes.  General  agreement. 

Specific  Comments : 

Yes,  I  hope  the  crews  take  this  material  to  heart  and  use 
it . 

Yes,  I  would  appreciate  being  included  in  the  studies. 
Yes,  it  would  help  identify  army  wide  program  weaknesses. 
It  might  be  better  to  track  a  class  from  flight  school. 
Would  be  interesting  as  participants  will  start  applying 
their  own  techniques.  This  training  is  just  the  base. 

Conclusion:  Recommended  that  the  class  roster  of  testbed 

participant  instructors  and  aircrews  be  retained  to  follow  up 
crew  coordination  policy,  program,  and  training  progress  over 
time . 


5.  What  effect  has  participation  in  this  project  had  on  you 
personally? 

Summary:  General  agreement  on  positive  effect. 

Specific  Comments : 

It  now  gives  me  something  to  grasp  so  that  I  can  put  it 
out  to  the  aviators. 

Regarding  techniques,  this  course  has  really  assisted. 
Improved  my  knowledge  of  scenario  development. 

Confirmed  previous  views  on  evaluating  task  performance. 
Clearly  defined  previous  beliefs  about  crew  coordination. 
Focuses  common  knowledge. 

Improved  my  ability  to  communicate  strengths  and 
weaknesses  during  evaluations. 

Makes  my  evaluations  more  objective  than  subjective. 

Able  to  evaluate  as  a  crew. 

Whether  this  program  is  fielded  or  not,  it's  made  me  a 
better  aviator  and  evaluator.  It's  given  me  another  tool 
to  use  to  teach  good  positive  habits  in  the  cockpit. 

It  supports  a  number  of  my  personal  beliefs  about  crew 
coordination. 

After  attending,  I  am  convinced  it  has  made  me  a  better 
crewmember . 

Anyone  who  has  the  desire  to  be  a  better  pilot  or 
crewmember  will  find  this  training  rewarding. 

I'm  already  using  the  concepts  taught  in  class  during 
missions  in  my  unit.  A  multiship  NVG  assault  mission 
last  week  was  the  best  mission  I  have  flown  (from 
planning  -  execution  -  AAR)  in  my  aviation  career. 

I  can  now  help  other  unit  aviators. 
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Makes  me  think  about  how  I  communicate  and  how  to  be  a 
better  communicator. 

I  never  knew  what  all  I  was  doing  wrong  until  I  attended 
this  course. 

Conclusion:  Recommend  that  the  letters  of  transmittal  that 
accompany  the  exportable  training  course  and  the  exportable 
evaluation  package  to  units  in  the  field  convey  the  positive 
effect  the  course  had  on  testbed  participants. 


6.  Do  you  have  any  questions  or  concerns  that  you  would  like  to 
ask  or  convey  to  the  crew  coordination  staff? 

Summary:  Open  discussion  surfaced  both  questions  and 
concerns . 

Specific  Comments : 

The  plan  for  flight  school  students.  Will  they  have  the 
benefit  of  videotaping  training?  [Yes,  funds  permitting. 
Probably  in  the  AQC  course.] 

Are  there  plans  to  monitor  IPs  on  the  flight  line?  [Yes, 
the  USAAVNC  has  already  started  with  the  Apache  IPs.] 

I  would  be  willing  to  work  on  exporting  this  package  on  a 
world-wide  basis.  I  believe  in  the  program. 

Extremely  professional.  It's  apparent  that  a  lot  of 
effort  and  thought  went  into  the  course. 

Major  concern  is  how  the  course  will  be  implemented  Army¬ 
wide  . 
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